Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..................................XX........................] duration: 3.019127507s FAIL (2025-08-17 04:39:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.019150461s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................XXXX....................] duration: 2.50004835s FAIL (2025-08-17 04:39:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.500077312s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....................................XXXXX..................] duration: 2.851488537s FAIL (2025-08-17 04:39:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.851550504s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................XXXX........................] duration: 2.779272914s FAIL (2025-08-17 04:39:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.779303297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................XXX.....................X....] duration: 2.828140813s FAIL (2025-08-17 04:39:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:36:48.495784617 (203h11m10.454881856s) [upload=211493]: Starting upload Aug 17 04:36:48.495860361 (203h11m10.454957607s) [upload=211493]: Done starting upload Aug 17 04:36:48.495936725 (203h11m10.455033963s) [upload=211493]: Uploading segments... Aug 17 04:36:48.496320051 (203h11m10.45541729s) [upload=211493]: Got next segment. Inline: false Aug 17 04:36:48.496456179 (203h11m10.455553431s) [upload=211493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:36:48.496464852 (203h11m10.455562087s) [upload=211493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:36:48.589685976 (203h11m10.54878323s) [upload=211493]: Next segment err: metaclient: Too Many Requests Aug 17 04:36:48.589713706 (203h11m10.548810973s) [upload=211493]: Done uploading segments... Aug 17 04:36:48.589733912 (203h11m10.548831143s) [upload=211493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.828246764s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................XXX.....................X...] duration: 2.384426977s FAIL (2025-08-17 04:38:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:36:48.495784617 (203h11m10.454881856s) [upload=211493]: Starting upload Aug 17 04:36:48.495860361 (203h11m10.454957607s) [upload=211493]: Done starting upload Aug 17 04:36:48.495936725 (203h11m10.455033963s) [upload=211493]: Uploading segments... Aug 17 04:36:48.496320051 (203h11m10.45541729s) [upload=211493]: Got next segment. Inline: false Aug 17 04:36:48.496456179 (203h11m10.455553431s) [upload=211493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:36:48.496464852 (203h11m10.455562087s) [upload=211493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:36:48.589685976 (203h11m10.54878323s) [upload=211493]: Next segment err: metaclient: Too Many Requests Aug 17 04:36:48.589713706 (203h11m10.548810973s) [upload=211493]: Done uploading segments... Aug 17 04:36:48.589733912 (203h11m10.548831143s) [upload=211493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.384474293s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...................................XX.......................] duration: 3.123574214s FAIL (2025-08-17 04:38:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.123600207s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................................XXXXX.................] duration: 2.721795437s FAIL (2025-08-17 04:38:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.721820051s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XXXX...................] duration: 2.303909542s FAIL (2025-08-17 04:38:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.303948146s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................XXX.....................X..] duration: 2.429862869s FAIL (2025-08-17 04:38:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:36:48.495784617 (203h11m10.454881856s) [upload=211493]: Starting upload Aug 17 04:36:48.495860361 (203h11m10.454957607s) [upload=211493]: Done starting upload Aug 17 04:36:48.495936725 (203h11m10.455033963s) [upload=211493]: Uploading segments... Aug 17 04:36:48.496320051 (203h11m10.45541729s) [upload=211493]: Got next segment. Inline: false Aug 17 04:36:48.496456179 (203h11m10.455553431s) [upload=211493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:36:48.496464852 (203h11m10.455562087s) [upload=211493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:36:48.589685976 (203h11m10.54878323s) [upload=211493]: Next segment err: metaclient: Too Many Requests Aug 17 04:36:48.589713706 (203h11m10.548810973s) [upload=211493]: Done uploading segments... Aug 17 04:36:48.589733912 (203h11m10.548831143s) [upload=211493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.429965109s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................XXXX.......................] duration: 3.166103928s FAIL (2025-08-17 04:38:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 3.166185155s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................XXXX..................] duration: 2.312436489s FAIL (2025-08-17 04:38:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.312489715s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [....................................XX......................] duration: 3.448038308s FAIL (2025-08-17 04:37:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.448066853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................XXX.....................X.] duration: 2.457211396s FAIL (2025-08-17 04:37:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:36:48.495784617 (203h11m10.454881856s) [upload=211493]: Starting upload Aug 17 04:36:48.495860361 (203h11m10.454957607s) [upload=211493]: Done starting upload Aug 17 04:36:48.495936725 (203h11m10.455033963s) [upload=211493]: Uploading segments... Aug 17 04:36:48.496320051 (203h11m10.45541729s) [upload=211493]: Got next segment. Inline: false Aug 17 04:36:48.496456179 (203h11m10.455553431s) [upload=211493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:36:48.496464852 (203h11m10.455562087s) [upload=211493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:36:48.589685976 (203h11m10.54878323s) [upload=211493]: Next segment err: metaclient: Too Many Requests Aug 17 04:36:48.589713706 (203h11m10.548810973s) [upload=211493]: Done uploading segments... Aug 17 04:36:48.589733912 (203h11m10.548831143s) [upload=211493]: Upload finished. err: metaclient: Too Many Requests Duration: 2.45726454s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......................................XXXXX................] duration: 2.832944451s FAIL (2025-08-17 04:37:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.833061939s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.....................................XX.....................] duration: 3.367158411s FAIL (2025-08-17 04:37:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.367181921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................XXXX......................] duration: 2.450848084s FAIL (2025-08-17 04:37:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.450885372s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XXXX.................] duration: 2.595125739s FAIL (2025-08-17 04:37:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.595158058s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................XXXX.....................] duration: 2.605810709s FAIL (2025-08-17 04:37:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.605846114s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [......................................XX....................] duration: 3.286667882s FAIL (2025-08-17 04:36:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.286685048s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................XXX.....................X] duration: 373.706099ms FAIL (2025-08-17 04:36:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 17 04:36:48.495784617 (203h11m10.454881856s) [upload=211493]: Starting upload Aug 17 04:36:48.495860361 (203h11m10.454957607s) [upload=211493]: Done starting upload Aug 17 04:36:48.495936725 (203h11m10.455033963s) [upload=211493]: Uploading segments... Aug 17 04:36:48.496320051 (203h11m10.45541729s) [upload=211493]: Got next segment. Inline: false Aug 17 04:36:48.496456179 (203h11m10.455553431s) [upload=211493]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:36:48.496464852 (203h11m10.455562087s) [upload=211493]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:36:48.589685976 (203h11m10.54878323s) [upload=211493]: Next segment err: metaclient: Too Many Requests Aug 17 04:36:48.589713706 (203h11m10.548810973s) [upload=211493]: Done uploading segments... Aug 17 04:36:48.589733912 (203h11m10.548831143s) [upload=211493]: Upload finished. err: metaclient: Too Many Requests Duration: 373.726746ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................XXXX....................] duration: 2.660920285s FAIL (2025-08-17 04:36:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.660952868s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........................................XXXXX...............] duration: 2.788520324s FAIL (2025-08-17 04:36:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.788549011s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................XXXX................] duration: 2.323997812s FAIL (2025-08-17 04:36:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.324032248s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XXXX...................] duration: 2.358543215s FAIL (2025-08-17 04:36:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.358577926s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XXXX...............] duration: 2.553116194s FAIL (2025-08-17 04:35:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.553152364s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.......................................XX...................] duration: 3.418328842s FAIL (2025-08-17 04:35:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.418353451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................XXX.....................] duration: 2.319004567s FAIL (2025-08-17 04:35:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 17 04:21:58.824378996 (202h56m20.783476241s) [upload=211227]: Starting upload Aug 17 04:21:58.824453728 (202h56m20.783550973s) [upload=211227]: Done starting upload Aug 17 04:21:58.824533712 (202h56m20.783630943s) [upload=211227]: Uploading segments... Aug 17 04:21:58.824814622 (202h56m20.783911863s) [upload=211227]: Got next segment. Inline: false Aug 17 04:21:58.824870724 (202h56m20.78396797s) [upload=211227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:21:58.824876461 (202h56m20.783973706s) [upload=211227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:21:59.025160521 (202h56m20.984257797s) [seg_pos={0 0}, upload=211227]: Begin upload segment... Aug 17 04:21:59.025238019 (202h56m20.984335266s) [seg_pos={0 0}, upload=211227]: Done begin upload segment. Aug 17 04:21:59.025273686 (202h56m20.98437092s) [noise=true , address=64.49.13.26:4951 , piece_id=SJYX5R2QSB64W6M4H5XJCPGSNF5YMJYBRFJF7FPGSZQB4KBAFO6A, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025331026 (202h56m20.984428309s) [noise=true , address=79.127.201.215:4357 , piece_id=5GN73I7DFN6XLU3D7VWZ7TTLYN2PQ2OAUVRRN4YOCZUUWL347EPQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025756251 (202h56m20.984853496s) [noise=true , address=45.91.116.12:4256 , piece_id=OITDESJQZQDVMAEJBQU4KAXWBY43JKXR7AHSE7OQ2TQKG5UKMQPQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025907425 (202h56m20.98500468s) [noise=true , address=104.36.82.35:4603 , piece_id=ZFG3KRVDGFTJPRYVOB7YFHAJDVS3FREBSOTHPZNHIAOOKNMAPJTA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026127117 (202h56m20.98522437s) [noise=true , address=77.90.31.14:4673 , piece_id=GANKYC2PWGNMZXLY7ZTB5RUFQLTFDZW3CH4AXQJ4FYD673I4WTWQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026350346 (202h56m20.985447594s) [noise=true , address=104.36.82.44:4901 , piece_id=QYKMZVHQ3PNWZ2NPWR4QQNDJWFPZW3PKUQIP6BA2YAQA55R6JWDA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026436670 (202h56m20.985533916s) [noise=true , address=77.90.31.19:4844 , piece_id=JKFXI6B5LZIIXLMF2UN3N226X3YX5ZRWPDJI6DYGRVZRAEOZEEIA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026714369 (202h56m20.985811614s) [noise=true , address=77.90.31.11:4580 , piece_id=TS52Z2GSFQMC722KXJCVYXNMW7UT73PGVWMQFBAXSJZE77CTP7UA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026788896 (202h56m20.985886144s) [noise=true , address=77.90.31.13:4641 , piece_id=SA6Z2Y3GPMFS2KSQNG2OX5QTEQ6UQMCSI7SAY2BR63NNOS37OTAA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027068637 (202h56m20.986165882s) [noise=true , address=79.127.220.101:4068 , piece_id=WHJJIXY6ZBZJNQQKJEML5XP5DMRJGUTDRFMOWWD6DJNDSUWCEI6A, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027120600 (202h56m20.986217846s) [noise=true , address=77.90.31.16:4745 , piece_id=SSPWQSSSZNDOSAVYSFO3WKTOH3MZT5WRW5S3ACYWYQGX54R7BDKQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027420110 (202h56m20.986517355s) [noise=true , address=104.36.82.60:4365 , piece_id=4BYPCTHJK6JQKEDQFTQJD6CYZQFO2GGB4WSX7GWTRC4BT6QS5JEQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027425372 (202h56m20.98652262s) [noise=true , address=77.90.31.10:4524 , piece_id=7KW7HUNUBCJWMG66YHQJ63H2UERBGQTIUV24U4WARUQWMHXEFSQQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027839122 (202h56m20.986936376s) [noise=true , address=66.23.193.175:4553 , piece_id=DQWBH4FPAYOHWQKATN7XPI3B5RFBZ4MF7WUMVS6FR7L5CXNW3JCQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.028247142 (202h56m20.987344396s) [noise=true , address=104.36.82.5:4866 , piece_id=D54LIPVYHKYLIVZRYBOMNOLVYSK3EDH7WDEZ5E563AQ35FUBELAQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.030005732 (202h56m20.989102986s) [noise=true , address=79.127.205.246:4275 , piece_id=ANNA5TM2H5ATR4ZCFZUW5Z2EFFKD6U2X2BWE7MB43RMPE4I7TADQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.030432900 (202h56m20.989530147s) [noise=true , address=45.91.116.11:4132 , piece_id=YPR6JXF4CAZBHP5XZUA5QIK2RFG75HHIRYSQ33EDTNSRT545YY7Q, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.031846491 (202h56m20.990943746s) [noise=true , address=104.36.82.23:4447 , piece_id=A56LTPVK6MNLOUCFJ7FXSAWFUOVQRLMUJB4PS5CYIEQTIWIG7DKQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.588920323 (202h56m21.548017569s) [upload=211227]: Next returned nil segment Aug 17 04:21:59.588944741 (202h56m21.548041972s) [upload=211227]: Waiting for error group managing segments... Aug 17 04:21:59.741262552 (202h56m21.700359791s) [noise=true , address=45.91.116.12:4256 , piece_id=OITDESJQZQDVMAEJBQU4KAXWBY43JKXR7AHSE7OQ2TQKG5UKMQPQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.742768290 (202h56m21.701865528s) [noise=true , address=77.90.31.11:4580 , piece_id=TS52Z2GSFQMC722KXJCVYXNMW7UT73PGVWMQFBAXSJZE77CTP7UA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.777622259 (202h56m21.736719503s) [noise=true , address=77.90.31.14:4673 , piece_id=GANKYC2PWGNMZXLY7ZTB5RUFQLTFDZW3CH4AXQJ4FYD673I4WTWQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.808692811 (202h56m21.767790057s) [noise=true , address=77.90.31.16:4745 , piece_id=SSPWQSSSZNDOSAVYSFO3WKTOH3MZT5WRW5S3ACYWYQGX54R7BDKQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.831688332 (202h56m21.790785579s) [noise=true , address=77.90.31.19:4844 , piece_id=JKFXI6B5LZIIXLMF2UN3N226X3YX5ZRWPDJI6DYGRVZRAEOZEEIA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.871745680 (202h56m21.830842918s) [noise=true , address=77.90.31.10:4524 , piece_id=7KW7HUNUBCJWMG66YHQJ63H2UERBGQTIUV24U4WARUQWMHXEFSQQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.883314367 (202h56m21.84241162s) [noise=true , address=77.90.31.13:4641 , piece_id=SA6Z2Y3GPMFS2KSQNG2OX5QTEQ6UQMCSI7SAY2BR63NNOS37OTAA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.890311438 (202h56m21.849408688s) [noise=true , address=79.127.220.101:4068 , piece_id=WHJJIXY6ZBZJNQQKJEML5XP5DMRJGUTDRFMOWWD6DJNDSUWCEI6A, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.940441979 (202h56m21.899539228s) [noise=true , address=45.91.116.11:4132 , piece_id=YPR6JXF4CAZBHP5XZUA5QIK2RFG75HHIRYSQ33EDTNSRT545YY7Q, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.976656973 (202h56m21.935754218s) [noise=true , address=79.127.201.215:4357 , piece_id=5GN73I7DFN6XLU3D7VWZ7TTLYN2PQ2OAUVRRN4YOCZUUWL347EPQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.283185203 (202h56m22.242282448s) [noise=true , address=66.23.193.175:4553 , piece_id=DQWBH4FPAYOHWQKATN7XPI3B5RFBZ4MF7WUMVS6FR7L5CXNW3JCQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.284822985 (202h56m22.243920238s) [noise=true , address=64.49.13.26:4951 , piece_id=SJYX5R2QSB64W6M4H5XJCPGSNF5YMJYBRFJF7FPGSZQB4KBAFO6A, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.446988508 (202h56m22.406085753s) [noise=true , address=79.127.205.246:4275 , piece_id=ANNA5TM2H5ATR4ZCFZUW5Z2EFFKD6U2X2BWE7MB43RMPE4I7TADQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:58.545488378 (202h57m20.504585625s) [noise=true , address=104.36.82.5:4866 , piece_id=D54LIPVYHKYLIVZRYBOMNOLVYSK3EDH7WDEZ5E563AQ35FUBELAQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:104.36.82.5:4866): protocol: write tcp 139.59.144.22:57798->104.36.82.5:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545674721 (202h57m20.504771972s) [noise=true , address=104.36.82.23:4447 , piece_id=A56LTPVK6MNLOUCFJ7FXSAWFUOVQRLMUJB4PS5CYIEQTIWIG7DKQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:104.36.82.23:4447): protocol: write tcp 139.59.144.22:57190->104.36.82.23:4447: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545824338 (202h57m20.504921585s) [noise=true , address=104.36.82.44:4901 , piece_id=QYKMZVHQ3PNWZ2NPWR4QQNDJWFPZW3PKUQIP6BA2YAQA55R6JWDA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: write tcp 139.59.144.22:54098->104.36.82.44:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545853626 (202h57m20.504950858s) [noise=true , address=104.36.82.35:4603 , piece_id=ZFG3KRVDGFTJPRYVOB7YFHAJDVS3FREBSOTHPZNHIAOOKNMAPJTA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:12ASAeTMBu4sY1GuZiSuwTXmhPksA1Te2BVZdcoE9QvpLAmcQX3, address:104.36.82.35:4603): protocol: write tcp 139.59.144.22:49164->104.36.82.35:4603: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545907146 (202h57m20.505004393s) [noise=true , address=104.36.82.60:4365 , piece_id=4BYPCTHJK6JQKEDQFTQJD6CYZQFO2GGB4WSX7GWTRC4BT6QS5JEQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: write tcp 139.59.144.22:43990->104.36.82.60:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545989950 (202h57m20.505087195s) [seg_pos={0 0}, upload=211227]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 04:22:58.646357515 (202h57m20.605454767s) [upload=211227]: Done uploading segments... Aug 17 04:22:58.646391150 (202h57m20.605488398s) [upload=211227]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.319039069s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........................................XXXXX..............] duration: 2.814096463s FAIL (2025-08-17 04:35:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.815035723s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................XXXX..................] duration: 2.401989071s FAIL (2025-08-17 04:35:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.402018282s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XXXX..............] duration: 2.479752864s FAIL (2025-08-17 04:35:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.479777526s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........................................XXXXX.............] duration: 2.67444463s FAIL (2025-08-17 04:35:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.674464541s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................XXX....................] duration: 2.342353898s FAIL (2025-08-17 04:34:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 17 04:21:58.824378996 (202h56m20.783476241s) [upload=211227]: Starting upload Aug 17 04:21:58.824453728 (202h56m20.783550973s) [upload=211227]: Done starting upload Aug 17 04:21:58.824533712 (202h56m20.783630943s) [upload=211227]: Uploading segments... Aug 17 04:21:58.824814622 (202h56m20.783911863s) [upload=211227]: Got next segment. Inline: false Aug 17 04:21:58.824870724 (202h56m20.78396797s) [upload=211227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:21:58.824876461 (202h56m20.783973706s) [upload=211227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:21:59.025160521 (202h56m20.984257797s) [seg_pos={0 0}, upload=211227]: Begin upload segment... Aug 17 04:21:59.025238019 (202h56m20.984335266s) [seg_pos={0 0}, upload=211227]: Done begin upload segment. Aug 17 04:21:59.025273686 (202h56m20.98437092s) [noise=true , address=64.49.13.26:4951 , piece_id=SJYX5R2QSB64W6M4H5XJCPGSNF5YMJYBRFJF7FPGSZQB4KBAFO6A, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025331026 (202h56m20.984428309s) [noise=true , address=79.127.201.215:4357 , piece_id=5GN73I7DFN6XLU3D7VWZ7TTLYN2PQ2OAUVRRN4YOCZUUWL347EPQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025756251 (202h56m20.984853496s) [noise=true , address=45.91.116.12:4256 , piece_id=OITDESJQZQDVMAEJBQU4KAXWBY43JKXR7AHSE7OQ2TQKG5UKMQPQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025907425 (202h56m20.98500468s) [noise=true , address=104.36.82.35:4603 , piece_id=ZFG3KRVDGFTJPRYVOB7YFHAJDVS3FREBSOTHPZNHIAOOKNMAPJTA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026127117 (202h56m20.98522437s) [noise=true , address=77.90.31.14:4673 , piece_id=GANKYC2PWGNMZXLY7ZTB5RUFQLTFDZW3CH4AXQJ4FYD673I4WTWQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026350346 (202h56m20.985447594s) [noise=true , address=104.36.82.44:4901 , piece_id=QYKMZVHQ3PNWZ2NPWR4QQNDJWFPZW3PKUQIP6BA2YAQA55R6JWDA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026436670 (202h56m20.985533916s) [noise=true , address=77.90.31.19:4844 , piece_id=JKFXI6B5LZIIXLMF2UN3N226X3YX5ZRWPDJI6DYGRVZRAEOZEEIA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026714369 (202h56m20.985811614s) [noise=true , address=77.90.31.11:4580 , piece_id=TS52Z2GSFQMC722KXJCVYXNMW7UT73PGVWMQFBAXSJZE77CTP7UA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026788896 (202h56m20.985886144s) [noise=true , address=77.90.31.13:4641 , piece_id=SA6Z2Y3GPMFS2KSQNG2OX5QTEQ6UQMCSI7SAY2BR63NNOS37OTAA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027068637 (202h56m20.986165882s) [noise=true , address=79.127.220.101:4068 , piece_id=WHJJIXY6ZBZJNQQKJEML5XP5DMRJGUTDRFMOWWD6DJNDSUWCEI6A, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027120600 (202h56m20.986217846s) [noise=true , address=77.90.31.16:4745 , piece_id=SSPWQSSSZNDOSAVYSFO3WKTOH3MZT5WRW5S3ACYWYQGX54R7BDKQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027420110 (202h56m20.986517355s) [noise=true , address=104.36.82.60:4365 , piece_id=4BYPCTHJK6JQKEDQFTQJD6CYZQFO2GGB4WSX7GWTRC4BT6QS5JEQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027425372 (202h56m20.98652262s) [noise=true , address=77.90.31.10:4524 , piece_id=7KW7HUNUBCJWMG66YHQJ63H2UERBGQTIUV24U4WARUQWMHXEFSQQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027839122 (202h56m20.986936376s) [noise=true , address=66.23.193.175:4553 , piece_id=DQWBH4FPAYOHWQKATN7XPI3B5RFBZ4MF7WUMVS6FR7L5CXNW3JCQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.028247142 (202h56m20.987344396s) [noise=true , address=104.36.82.5:4866 , piece_id=D54LIPVYHKYLIVZRYBOMNOLVYSK3EDH7WDEZ5E563AQ35FUBELAQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.030005732 (202h56m20.989102986s) [noise=true , address=79.127.205.246:4275 , piece_id=ANNA5TM2H5ATR4ZCFZUW5Z2EFFKD6U2X2BWE7MB43RMPE4I7TADQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.030432900 (202h56m20.989530147s) [noise=true , address=45.91.116.11:4132 , piece_id=YPR6JXF4CAZBHP5XZUA5QIK2RFG75HHIRYSQ33EDTNSRT545YY7Q, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.031846491 (202h56m20.990943746s) [noise=true , address=104.36.82.23:4447 , piece_id=A56LTPVK6MNLOUCFJ7FXSAWFUOVQRLMUJB4PS5CYIEQTIWIG7DKQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.588920323 (202h56m21.548017569s) [upload=211227]: Next returned nil segment Aug 17 04:21:59.588944741 (202h56m21.548041972s) [upload=211227]: Waiting for error group managing segments... Aug 17 04:21:59.741262552 (202h56m21.700359791s) [noise=true , address=45.91.116.12:4256 , piece_id=OITDESJQZQDVMAEJBQU4KAXWBY43JKXR7AHSE7OQ2TQKG5UKMQPQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.742768290 (202h56m21.701865528s) [noise=true , address=77.90.31.11:4580 , piece_id=TS52Z2GSFQMC722KXJCVYXNMW7UT73PGVWMQFBAXSJZE77CTP7UA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.777622259 (202h56m21.736719503s) [noise=true , address=77.90.31.14:4673 , piece_id=GANKYC2PWGNMZXLY7ZTB5RUFQLTFDZW3CH4AXQJ4FYD673I4WTWQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.808692811 (202h56m21.767790057s) [noise=true , address=77.90.31.16:4745 , piece_id=SSPWQSSSZNDOSAVYSFO3WKTOH3MZT5WRW5S3ACYWYQGX54R7BDKQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.831688332 (202h56m21.790785579s) [noise=true , address=77.90.31.19:4844 , piece_id=JKFXI6B5LZIIXLMF2UN3N226X3YX5ZRWPDJI6DYGRVZRAEOZEEIA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.871745680 (202h56m21.830842918s) [noise=true , address=77.90.31.10:4524 , piece_id=7KW7HUNUBCJWMG66YHQJ63H2UERBGQTIUV24U4WARUQWMHXEFSQQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.883314367 (202h56m21.84241162s) [noise=true , address=77.90.31.13:4641 , piece_id=SA6Z2Y3GPMFS2KSQNG2OX5QTEQ6UQMCSI7SAY2BR63NNOS37OTAA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.890311438 (202h56m21.849408688s) [noise=true , address=79.127.220.101:4068 , piece_id=WHJJIXY6ZBZJNQQKJEML5XP5DMRJGUTDRFMOWWD6DJNDSUWCEI6A, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.940441979 (202h56m21.899539228s) [noise=true , address=45.91.116.11:4132 , piece_id=YPR6JXF4CAZBHP5XZUA5QIK2RFG75HHIRYSQ33EDTNSRT545YY7Q, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.976656973 (202h56m21.935754218s) [noise=true , address=79.127.201.215:4357 , piece_id=5GN73I7DFN6XLU3D7VWZ7TTLYN2PQ2OAUVRRN4YOCZUUWL347EPQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.283185203 (202h56m22.242282448s) [noise=true , address=66.23.193.175:4553 , piece_id=DQWBH4FPAYOHWQKATN7XPI3B5RFBZ4MF7WUMVS6FR7L5CXNW3JCQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.284822985 (202h56m22.243920238s) [noise=true , address=64.49.13.26:4951 , piece_id=SJYX5R2QSB64W6M4H5XJCPGSNF5YMJYBRFJF7FPGSZQB4KBAFO6A, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.446988508 (202h56m22.406085753s) [noise=true , address=79.127.205.246:4275 , piece_id=ANNA5TM2H5ATR4ZCFZUW5Z2EFFKD6U2X2BWE7MB43RMPE4I7TADQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:58.545488378 (202h57m20.504585625s) [noise=true , address=104.36.82.5:4866 , piece_id=D54LIPVYHKYLIVZRYBOMNOLVYSK3EDH7WDEZ5E563AQ35FUBELAQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:104.36.82.5:4866): protocol: write tcp 139.59.144.22:57798->104.36.82.5:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545674721 (202h57m20.504771972s) [noise=true , address=104.36.82.23:4447 , piece_id=A56LTPVK6MNLOUCFJ7FXSAWFUOVQRLMUJB4PS5CYIEQTIWIG7DKQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:104.36.82.23:4447): protocol: write tcp 139.59.144.22:57190->104.36.82.23:4447: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545824338 (202h57m20.504921585s) [noise=true , address=104.36.82.44:4901 , piece_id=QYKMZVHQ3PNWZ2NPWR4QQNDJWFPZW3PKUQIP6BA2YAQA55R6JWDA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: write tcp 139.59.144.22:54098->104.36.82.44:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545853626 (202h57m20.504950858s) [noise=true , address=104.36.82.35:4603 , piece_id=ZFG3KRVDGFTJPRYVOB7YFHAJDVS3FREBSOTHPZNHIAOOKNMAPJTA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:12ASAeTMBu4sY1GuZiSuwTXmhPksA1Te2BVZdcoE9QvpLAmcQX3, address:104.36.82.35:4603): protocol: write tcp 139.59.144.22:49164->104.36.82.35:4603: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545907146 (202h57m20.505004393s) [noise=true , address=104.36.82.60:4365 , piece_id=4BYPCTHJK6JQKEDQFTQJD6CYZQFO2GGB4WSX7GWTRC4BT6QS5JEQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: write tcp 139.59.144.22:43990->104.36.82.60:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545989950 (202h57m20.505087195s) [seg_pos={0 0}, upload=211227]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 04:22:58.646357515 (202h57m20.605454767s) [upload=211227]: Done uploading segments... Aug 17 04:22:58.646391150 (202h57m20.605488398s) [upload=211227]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.342399557s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XXXX.................] duration: 2.427201856s FAIL (2025-08-17 04:34:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.427234738s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [........................................XX..................] duration: 2.984095658s FAIL (2025-08-17 04:34:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.984129708s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................................XXXXX............] duration: 2.953400558s FAIL (2025-08-17 04:34:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.953424452s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................XX.................] duration: 2.977243451s FAIL (2025-08-17 04:34:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.97731749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XXXX.............] duration: 2.288376593s FAIL (2025-08-17 04:34:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.288421212s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................XXX...................] duration: 2.432280569s FAIL (2025-08-17 04:34:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 17 04:21:58.824378996 (202h56m20.783476241s) [upload=211227]: Starting upload Aug 17 04:21:58.824453728 (202h56m20.783550973s) [upload=211227]: Done starting upload Aug 17 04:21:58.824533712 (202h56m20.783630943s) [upload=211227]: Uploading segments... Aug 17 04:21:58.824814622 (202h56m20.783911863s) [upload=211227]: Got next segment. Inline: false Aug 17 04:21:58.824870724 (202h56m20.78396797s) [upload=211227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:21:58.824876461 (202h56m20.783973706s) [upload=211227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:21:59.025160521 (202h56m20.984257797s) [seg_pos={0 0}, upload=211227]: Begin upload segment... Aug 17 04:21:59.025238019 (202h56m20.984335266s) [seg_pos={0 0}, upload=211227]: Done begin upload segment. Aug 17 04:21:59.025273686 (202h56m20.98437092s) [noise=true , address=64.49.13.26:4951 , piece_id=SJYX5R2QSB64W6M4H5XJCPGSNF5YMJYBRFJF7FPGSZQB4KBAFO6A, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025331026 (202h56m20.984428309s) [noise=true , address=79.127.201.215:4357 , piece_id=5GN73I7DFN6XLU3D7VWZ7TTLYN2PQ2OAUVRRN4YOCZUUWL347EPQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025756251 (202h56m20.984853496s) [noise=true , address=45.91.116.12:4256 , piece_id=OITDESJQZQDVMAEJBQU4KAXWBY43JKXR7AHSE7OQ2TQKG5UKMQPQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025907425 (202h56m20.98500468s) [noise=true , address=104.36.82.35:4603 , piece_id=ZFG3KRVDGFTJPRYVOB7YFHAJDVS3FREBSOTHPZNHIAOOKNMAPJTA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026127117 (202h56m20.98522437s) [noise=true , address=77.90.31.14:4673 , piece_id=GANKYC2PWGNMZXLY7ZTB5RUFQLTFDZW3CH4AXQJ4FYD673I4WTWQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026350346 (202h56m20.985447594s) [noise=true , address=104.36.82.44:4901 , piece_id=QYKMZVHQ3PNWZ2NPWR4QQNDJWFPZW3PKUQIP6BA2YAQA55R6JWDA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026436670 (202h56m20.985533916s) [noise=true , address=77.90.31.19:4844 , piece_id=JKFXI6B5LZIIXLMF2UN3N226X3YX5ZRWPDJI6DYGRVZRAEOZEEIA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026714369 (202h56m20.985811614s) [noise=true , address=77.90.31.11:4580 , piece_id=TS52Z2GSFQMC722KXJCVYXNMW7UT73PGVWMQFBAXSJZE77CTP7UA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026788896 (202h56m20.985886144s) [noise=true , address=77.90.31.13:4641 , piece_id=SA6Z2Y3GPMFS2KSQNG2OX5QTEQ6UQMCSI7SAY2BR63NNOS37OTAA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027068637 (202h56m20.986165882s) [noise=true , address=79.127.220.101:4068 , piece_id=WHJJIXY6ZBZJNQQKJEML5XP5DMRJGUTDRFMOWWD6DJNDSUWCEI6A, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027120600 (202h56m20.986217846s) [noise=true , address=77.90.31.16:4745 , piece_id=SSPWQSSSZNDOSAVYSFO3WKTOH3MZT5WRW5S3ACYWYQGX54R7BDKQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027420110 (202h56m20.986517355s) [noise=true , address=104.36.82.60:4365 , piece_id=4BYPCTHJK6JQKEDQFTQJD6CYZQFO2GGB4WSX7GWTRC4BT6QS5JEQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027425372 (202h56m20.98652262s) [noise=true , address=77.90.31.10:4524 , piece_id=7KW7HUNUBCJWMG66YHQJ63H2UERBGQTIUV24U4WARUQWMHXEFSQQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027839122 (202h56m20.986936376s) [noise=true , address=66.23.193.175:4553 , piece_id=DQWBH4FPAYOHWQKATN7XPI3B5RFBZ4MF7WUMVS6FR7L5CXNW3JCQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.028247142 (202h56m20.987344396s) [noise=true , address=104.36.82.5:4866 , piece_id=D54LIPVYHKYLIVZRYBOMNOLVYSK3EDH7WDEZ5E563AQ35FUBELAQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.030005732 (202h56m20.989102986s) [noise=true , address=79.127.205.246:4275 , piece_id=ANNA5TM2H5ATR4ZCFZUW5Z2EFFKD6U2X2BWE7MB43RMPE4I7TADQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.030432900 (202h56m20.989530147s) [noise=true , address=45.91.116.11:4132 , piece_id=YPR6JXF4CAZBHP5XZUA5QIK2RFG75HHIRYSQ33EDTNSRT545YY7Q, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.031846491 (202h56m20.990943746s) [noise=true , address=104.36.82.23:4447 , piece_id=A56LTPVK6MNLOUCFJ7FXSAWFUOVQRLMUJB4PS5CYIEQTIWIG7DKQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.588920323 (202h56m21.548017569s) [upload=211227]: Next returned nil segment Aug 17 04:21:59.588944741 (202h56m21.548041972s) [upload=211227]: Waiting for error group managing segments... Aug 17 04:21:59.741262552 (202h56m21.700359791s) [noise=true , address=45.91.116.12:4256 , piece_id=OITDESJQZQDVMAEJBQU4KAXWBY43JKXR7AHSE7OQ2TQKG5UKMQPQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.742768290 (202h56m21.701865528s) [noise=true , address=77.90.31.11:4580 , piece_id=TS52Z2GSFQMC722KXJCVYXNMW7UT73PGVWMQFBAXSJZE77CTP7UA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.777622259 (202h56m21.736719503s) [noise=true , address=77.90.31.14:4673 , piece_id=GANKYC2PWGNMZXLY7ZTB5RUFQLTFDZW3CH4AXQJ4FYD673I4WTWQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.808692811 (202h56m21.767790057s) [noise=true , address=77.90.31.16:4745 , piece_id=SSPWQSSSZNDOSAVYSFO3WKTOH3MZT5WRW5S3ACYWYQGX54R7BDKQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.831688332 (202h56m21.790785579s) [noise=true , address=77.90.31.19:4844 , piece_id=JKFXI6B5LZIIXLMF2UN3N226X3YX5ZRWPDJI6DYGRVZRAEOZEEIA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.871745680 (202h56m21.830842918s) [noise=true , address=77.90.31.10:4524 , piece_id=7KW7HUNUBCJWMG66YHQJ63H2UERBGQTIUV24U4WARUQWMHXEFSQQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.883314367 (202h56m21.84241162s) [noise=true , address=77.90.31.13:4641 , piece_id=SA6Z2Y3GPMFS2KSQNG2OX5QTEQ6UQMCSI7SAY2BR63NNOS37OTAA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.890311438 (202h56m21.849408688s) [noise=true , address=79.127.220.101:4068 , piece_id=WHJJIXY6ZBZJNQQKJEML5XP5DMRJGUTDRFMOWWD6DJNDSUWCEI6A, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.940441979 (202h56m21.899539228s) [noise=true , address=45.91.116.11:4132 , piece_id=YPR6JXF4CAZBHP5XZUA5QIK2RFG75HHIRYSQ33EDTNSRT545YY7Q, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.976656973 (202h56m21.935754218s) [noise=true , address=79.127.201.215:4357 , piece_id=5GN73I7DFN6XLU3D7VWZ7TTLYN2PQ2OAUVRRN4YOCZUUWL347EPQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.283185203 (202h56m22.242282448s) [noise=true , address=66.23.193.175:4553 , piece_id=DQWBH4FPAYOHWQKATN7XPI3B5RFBZ4MF7WUMVS6FR7L5CXNW3JCQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.284822985 (202h56m22.243920238s) [noise=true , address=64.49.13.26:4951 , piece_id=SJYX5R2QSB64W6M4H5XJCPGSNF5YMJYBRFJF7FPGSZQB4KBAFO6A, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.446988508 (202h56m22.406085753s) [noise=true , address=79.127.205.246:4275 , piece_id=ANNA5TM2H5ATR4ZCFZUW5Z2EFFKD6U2X2BWE7MB43RMPE4I7TADQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:58.545488378 (202h57m20.504585625s) [noise=true , address=104.36.82.5:4866 , piece_id=D54LIPVYHKYLIVZRYBOMNOLVYSK3EDH7WDEZ5E563AQ35FUBELAQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:104.36.82.5:4866): protocol: write tcp 139.59.144.22:57798->104.36.82.5:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545674721 (202h57m20.504771972s) [noise=true , address=104.36.82.23:4447 , piece_id=A56LTPVK6MNLOUCFJ7FXSAWFUOVQRLMUJB4PS5CYIEQTIWIG7DKQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:104.36.82.23:4447): protocol: write tcp 139.59.144.22:57190->104.36.82.23:4447: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545824338 (202h57m20.504921585s) [noise=true , address=104.36.82.44:4901 , piece_id=QYKMZVHQ3PNWZ2NPWR4QQNDJWFPZW3PKUQIP6BA2YAQA55R6JWDA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: write tcp 139.59.144.22:54098->104.36.82.44:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545853626 (202h57m20.504950858s) [noise=true , address=104.36.82.35:4603 , piece_id=ZFG3KRVDGFTJPRYVOB7YFHAJDVS3FREBSOTHPZNHIAOOKNMAPJTA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:12ASAeTMBu4sY1GuZiSuwTXmhPksA1Te2BVZdcoE9QvpLAmcQX3, address:104.36.82.35:4603): protocol: write tcp 139.59.144.22:49164->104.36.82.35:4603: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545907146 (202h57m20.505004393s) [noise=true , address=104.36.82.60:4365 , piece_id=4BYPCTHJK6JQKEDQFTQJD6CYZQFO2GGB4WSX7GWTRC4BT6QS5JEQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: write tcp 139.59.144.22:43990->104.36.82.60:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545989950 (202h57m20.505087195s) [seg_pos={0 0}, upload=211227]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 04:22:58.646357515 (202h57m20.605454767s) [upload=211227]: Done uploading segments... Aug 17 04:22:58.646391150 (202h57m20.605488398s) [upload=211227]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.432322551s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................XXXX............] duration: 2.612725526s FAIL (2025-08-17 04:33:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 17 04:22:36.193175039 (202h56m58.152272297s) [upload=211241]: Starting upload Aug 17 04:22:36.193282733 (202h56m58.152379987s) [upload=211241]: Done starting upload Aug 17 04:22:36.193533603 (202h56m58.152630849s) [upload=211241]: Uploading segments... Aug 17 04:22:36.193824638 (202h56m58.152921893s) [upload=211241]: Got next segment. Inline: false Aug 17 04:22:36.193891646 (202h56m58.152988897s) [upload=211241]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:22:36.193898487 (202h56m58.15299574s) [upload=211241]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:22:36.387313731 (202h56m58.346410984s) [seg_pos={0 0}, upload=211241]: Begin upload segment... Aug 17 04:22:36.387391508 (202h56m58.346488754s) [seg_pos={0 0}, upload=211241]: Done begin upload segment. Aug 17 04:22:36.387453494 (202h56m58.34655085s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.387549035 (202h56m58.346646289s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388292176 (202h56m58.347389421s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388564570 (202h56m58.347661824s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388775483 (202h56m58.347872728s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.388976887 (202h56m58.348074132s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389048807 (202h56m58.348146055s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389269495 (202h56m58.348366743s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389303339 (202h56m58.348400583s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389577693 (202h56m58.348674943s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389582267 (202h56m58.348679515s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389868029 (202h56m58.348965273s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.389899606 (202h56m58.348996853s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390181444 (202h56m58.349278692s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.390456412 (202h56m58.349553657s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:36.396850594 (202h56m58.355947842s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Uploading piece... Aug 17 04:22:37.003486038 (202h56m58.962583287s) [upload=211241]: Next returned nil segment Aug 17 04:22:37.003509888 (202h56m58.962607148s) [upload=211241]: Waiting for error group managing segments... Aug 17 04:22:37.141087302 (202h56m59.100184557s) [noise=true , address=45.91.116.4:4939 , piece_id=INXRARFNJD2KCP6IDBOCVAZA5U4QJX5TVPQALFXJZ5V6YNOGG5FQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.147724589 (202h56m59.106821827s) [noise=true , address=45.91.116.14:4190 , piece_id=RMPPQYC3JOXNR25LRTNKUHHVXSLWRYS4UK3WFL7JUN2IEB7MTPFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148191395 (202h56m59.107288641s) [noise=true , address=45.91.116.6:4974 , piece_id=SG5PHVVY6ZRIJLCQX6PW4BGKNNETDF3TWGM74GDYNJQUXPCG5DOA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.148249179 (202h56m59.107346437s) [noise=true , address=45.91.116.13:4153 , piece_id=WDOI5HF6IFTRHNWSZS7OCRVHJFWIKLOIUFKIO6HIA6DDD65V5AFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.154197920 (202h56m59.113295159s) [noise=true , address=45.91.116.11:4119 , piece_id=RTDAZ627L753OYJZNDWQEXNQKWNEOO4O6NXKVCAQJOFDR5O22AJQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155088755 (202h56m59.11418599s) [noise=true , address=45.91.116.9:4261 , piece_id=5WVHJ2XXGMLB5EFKDSI6XJD27MKZ3H76DR7M6NE4D4PUCNHZHU5A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.155737855 (202h56m59.114835102s) [noise=true , address=45.91.116.12:4226 , piece_id=FX6QBTY2YZK4RWYQQJICS23MPPGMPQY6CX3AEXKDJ26UQ35AIDVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.158542805 (202h56m59.117640044s) [noise=true , address=77.90.31.18:4807 , piece_id=PWOXWGYMFXZV55YHFVBSUTHPSUF2K5FEJADLMFWTICIRBXJGNW2A, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.163837059 (202h56m59.122934309s) [noise=true , address=45.91.116.5:4297 , piece_id=BKI5XCEXSAE4545PWGGL3RFHM5M5ZGBCAKLJYX2JFIUGJBC3LKXA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.225669823 (202h56m59.184767059s) [noise=true , address=45.91.116.7:4009 , piece_id=BQNCWPP6OVPYYTAMJPJCWACNVYW6BCSFDUH5GY7LRSYWHILWPULQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:22:37.249261694 (202h56m59.20835894s) [noise=true , address=77.90.31.14:4663 , piece_id=QHGIW4YTHWRUZC2GAJVRMTHMSTBMZTFSIWDXCQ72U5PHTPE6HDFA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: <nil> Aug 17 04:23:05.920185881 (202h57m27.879283118s) [noise=true , address=104.36.82.10:4079 , piece_id=JCA4FKNW3R7YPGU2FGO7OXK3JPI4J2NCSEYJNLXLX6EW6B4LNNSQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920277695 (202h57m27.879374941s) [noise=true , address=104.36.82.15:4259 , piece_id=L5C3UKGCVNKD52DSBG7OT53ON5NNG27WARZXUVK4DGRI5X2YITVA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920476811 (202h57m27.87957406s) [noise=true , address=104.36.82.7:4911 , piece_id=OL5RYLAFZM7YOJQXANQU5LIVSU7IFGN6WKB2N2TCRXHQGOILZ7JQ, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920550680 (202h57m27.879647915s) [noise=true , address=104.36.82.4:4801 , piece_id=RBMJ6IZSFMGIOZN3256PTIGBTYIQFYQFZFPHPFOZXNRTB5W3Z6HA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:05.920687798 (202h57m27.879785037s) [noise=true , address=104.36.82.8:4971 , piece_id=UNRIC3GXTSOWM266PVJYN3M33P474DKTAFTRQGJYYYV2VG2UH3YA, seg_pos={0 0}, upload=211241]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:52446->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 17 04:23:05.920736972 (202h57m27.879834217s) [seg_pos={0 0}, upload=211241]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 17 04:23:06.024578477 (202h57m27.983675715s) [upload=211241]: Done uploading segments... Aug 17 04:23:06.024615107 (202h57m27.983712354s) [upload=211241]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.612759233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................XXXX................] duration: 2.750201196s FAIL (2025-08-17 04:33:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.750234454s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................XX................] duration: 2.964873363s FAIL (2025-08-17 04:33:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.964914571s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................................XXXXX...........] duration: 2.739574785s FAIL (2025-08-17 04:33:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.739618136s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XXXX...............] duration: 2.494559247s FAIL (2025-08-17 04:33:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.494589609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...........................................XX...............] duration: 3.03905232s FAIL (2025-08-17 04:33:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.039075841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................XXX..................] duration: 2.862679053s FAIL (2025-08-17 04:33:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 17 04:21:58.824378996 (202h56m20.783476241s) [upload=211227]: Starting upload Aug 17 04:21:58.824453728 (202h56m20.783550973s) [upload=211227]: Done starting upload Aug 17 04:21:58.824533712 (202h56m20.783630943s) [upload=211227]: Uploading segments... Aug 17 04:21:58.824814622 (202h56m20.783911863s) [upload=211227]: Got next segment. Inline: false Aug 17 04:21:58.824870724 (202h56m20.78396797s) [upload=211227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:21:58.824876461 (202h56m20.783973706s) [upload=211227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:21:59.025160521 (202h56m20.984257797s) [seg_pos={0 0}, upload=211227]: Begin upload segment... Aug 17 04:21:59.025238019 (202h56m20.984335266s) [seg_pos={0 0}, upload=211227]: Done begin upload segment. Aug 17 04:21:59.025273686 (202h56m20.98437092s) [noise=true , address=64.49.13.26:4951 , piece_id=SJYX5R2QSB64W6M4H5XJCPGSNF5YMJYBRFJF7FPGSZQB4KBAFO6A, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025331026 (202h56m20.984428309s) [noise=true , address=79.127.201.215:4357 , piece_id=5GN73I7DFN6XLU3D7VWZ7TTLYN2PQ2OAUVRRN4YOCZUUWL347EPQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025756251 (202h56m20.984853496s) [noise=true , address=45.91.116.12:4256 , piece_id=OITDESJQZQDVMAEJBQU4KAXWBY43JKXR7AHSE7OQ2TQKG5UKMQPQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025907425 (202h56m20.98500468s) [noise=true , address=104.36.82.35:4603 , piece_id=ZFG3KRVDGFTJPRYVOB7YFHAJDVS3FREBSOTHPZNHIAOOKNMAPJTA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026127117 (202h56m20.98522437s) [noise=true , address=77.90.31.14:4673 , piece_id=GANKYC2PWGNMZXLY7ZTB5RUFQLTFDZW3CH4AXQJ4FYD673I4WTWQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026350346 (202h56m20.985447594s) [noise=true , address=104.36.82.44:4901 , piece_id=QYKMZVHQ3PNWZ2NPWR4QQNDJWFPZW3PKUQIP6BA2YAQA55R6JWDA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026436670 (202h56m20.985533916s) [noise=true , address=77.90.31.19:4844 , piece_id=JKFXI6B5LZIIXLMF2UN3N226X3YX5ZRWPDJI6DYGRVZRAEOZEEIA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026714369 (202h56m20.985811614s) [noise=true , address=77.90.31.11:4580 , piece_id=TS52Z2GSFQMC722KXJCVYXNMW7UT73PGVWMQFBAXSJZE77CTP7UA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026788896 (202h56m20.985886144s) [noise=true , address=77.90.31.13:4641 , piece_id=SA6Z2Y3GPMFS2KSQNG2OX5QTEQ6UQMCSI7SAY2BR63NNOS37OTAA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027068637 (202h56m20.986165882s) [noise=true , address=79.127.220.101:4068 , piece_id=WHJJIXY6ZBZJNQQKJEML5XP5DMRJGUTDRFMOWWD6DJNDSUWCEI6A, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027120600 (202h56m20.986217846s) [noise=true , address=77.90.31.16:4745 , piece_id=SSPWQSSSZNDOSAVYSFO3WKTOH3MZT5WRW5S3ACYWYQGX54R7BDKQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027420110 (202h56m20.986517355s) [noise=true , address=104.36.82.60:4365 , piece_id=4BYPCTHJK6JQKEDQFTQJD6CYZQFO2GGB4WSX7GWTRC4BT6QS5JEQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027425372 (202h56m20.98652262s) [noise=true , address=77.90.31.10:4524 , piece_id=7KW7HUNUBCJWMG66YHQJ63H2UERBGQTIUV24U4WARUQWMHXEFSQQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027839122 (202h56m20.986936376s) [noise=true , address=66.23.193.175:4553 , piece_id=DQWBH4FPAYOHWQKATN7XPI3B5RFBZ4MF7WUMVS6FR7L5CXNW3JCQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.028247142 (202h56m20.987344396s) [noise=true , address=104.36.82.5:4866 , piece_id=D54LIPVYHKYLIVZRYBOMNOLVYSK3EDH7WDEZ5E563AQ35FUBELAQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.030005732 (202h56m20.989102986s) [noise=true , address=79.127.205.246:4275 , piece_id=ANNA5TM2H5ATR4ZCFZUW5Z2EFFKD6U2X2BWE7MB43RMPE4I7TADQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.030432900 (202h56m20.989530147s) [noise=true , address=45.91.116.11:4132 , piece_id=YPR6JXF4CAZBHP5XZUA5QIK2RFG75HHIRYSQ33EDTNSRT545YY7Q, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.031846491 (202h56m20.990943746s) [noise=true , address=104.36.82.23:4447 , piece_id=A56LTPVK6MNLOUCFJ7FXSAWFUOVQRLMUJB4PS5CYIEQTIWIG7DKQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.588920323 (202h56m21.548017569s) [upload=211227]: Next returned nil segment Aug 17 04:21:59.588944741 (202h56m21.548041972s) [upload=211227]: Waiting for error group managing segments... Aug 17 04:21:59.741262552 (202h56m21.700359791s) [noise=true , address=45.91.116.12:4256 , piece_id=OITDESJQZQDVMAEJBQU4KAXWBY43JKXR7AHSE7OQ2TQKG5UKMQPQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.742768290 (202h56m21.701865528s) [noise=true , address=77.90.31.11:4580 , piece_id=TS52Z2GSFQMC722KXJCVYXNMW7UT73PGVWMQFBAXSJZE77CTP7UA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.777622259 (202h56m21.736719503s) [noise=true , address=77.90.31.14:4673 , piece_id=GANKYC2PWGNMZXLY7ZTB5RUFQLTFDZW3CH4AXQJ4FYD673I4WTWQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.808692811 (202h56m21.767790057s) [noise=true , address=77.90.31.16:4745 , piece_id=SSPWQSSSZNDOSAVYSFO3WKTOH3MZT5WRW5S3ACYWYQGX54R7BDKQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.831688332 (202h56m21.790785579s) [noise=true , address=77.90.31.19:4844 , piece_id=JKFXI6B5LZIIXLMF2UN3N226X3YX5ZRWPDJI6DYGRVZRAEOZEEIA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.871745680 (202h56m21.830842918s) [noise=true , address=77.90.31.10:4524 , piece_id=7KW7HUNUBCJWMG66YHQJ63H2UERBGQTIUV24U4WARUQWMHXEFSQQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.883314367 (202h56m21.84241162s) [noise=true , address=77.90.31.13:4641 , piece_id=SA6Z2Y3GPMFS2KSQNG2OX5QTEQ6UQMCSI7SAY2BR63NNOS37OTAA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.890311438 (202h56m21.849408688s) [noise=true , address=79.127.220.101:4068 , piece_id=WHJJIXY6ZBZJNQQKJEML5XP5DMRJGUTDRFMOWWD6DJNDSUWCEI6A, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.940441979 (202h56m21.899539228s) [noise=true , address=45.91.116.11:4132 , piece_id=YPR6JXF4CAZBHP5XZUA5QIK2RFG75HHIRYSQ33EDTNSRT545YY7Q, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.976656973 (202h56m21.935754218s) [noise=true , address=79.127.201.215:4357 , piece_id=5GN73I7DFN6XLU3D7VWZ7TTLYN2PQ2OAUVRRN4YOCZUUWL347EPQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.283185203 (202h56m22.242282448s) [noise=true , address=66.23.193.175:4553 , piece_id=DQWBH4FPAYOHWQKATN7XPI3B5RFBZ4MF7WUMVS6FR7L5CXNW3JCQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.284822985 (202h56m22.243920238s) [noise=true , address=64.49.13.26:4951 , piece_id=SJYX5R2QSB64W6M4H5XJCPGSNF5YMJYBRFJF7FPGSZQB4KBAFO6A, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.446988508 (202h56m22.406085753s) [noise=true , address=79.127.205.246:4275 , piece_id=ANNA5TM2H5ATR4ZCFZUW5Z2EFFKD6U2X2BWE7MB43RMPE4I7TADQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:58.545488378 (202h57m20.504585625s) [noise=true , address=104.36.82.5:4866 , piece_id=D54LIPVYHKYLIVZRYBOMNOLVYSK3EDH7WDEZ5E563AQ35FUBELAQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:104.36.82.5:4866): protocol: write tcp 139.59.144.22:57798->104.36.82.5:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545674721 (202h57m20.504771972s) [noise=true , address=104.36.82.23:4447 , piece_id=A56LTPVK6MNLOUCFJ7FXSAWFUOVQRLMUJB4PS5CYIEQTIWIG7DKQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:104.36.82.23:4447): protocol: write tcp 139.59.144.22:57190->104.36.82.23:4447: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545824338 (202h57m20.504921585s) [noise=true , address=104.36.82.44:4901 , piece_id=QYKMZVHQ3PNWZ2NPWR4QQNDJWFPZW3PKUQIP6BA2YAQA55R6JWDA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: write tcp 139.59.144.22:54098->104.36.82.44:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545853626 (202h57m20.504950858s) [noise=true , address=104.36.82.35:4603 , piece_id=ZFG3KRVDGFTJPRYVOB7YFHAJDVS3FREBSOTHPZNHIAOOKNMAPJTA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:12ASAeTMBu4sY1GuZiSuwTXmhPksA1Te2BVZdcoE9QvpLAmcQX3, address:104.36.82.35:4603): protocol: write tcp 139.59.144.22:49164->104.36.82.35:4603: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545907146 (202h57m20.505004393s) [noise=true , address=104.36.82.60:4365 , piece_id=4BYPCTHJK6JQKEDQFTQJD6CYZQFO2GGB4WSX7GWTRC4BT6QS5JEQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: write tcp 139.59.144.22:43990->104.36.82.60:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545989950 (202h57m20.505087195s) [seg_pos={0 0}, upload=211227]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 04:22:58.646357515 (202h57m20.605454767s) [upload=211227]: Done uploading segments... Aug 17 04:22:58.646391150 (202h57m20.605488398s) [upload=211227]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.862737358s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XXXX..............] duration: 2.336782436s FAIL (2025-08-17 04:32:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.336818039s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................................XXXXX..........] duration: 2.63757804s FAIL (2025-08-17 04:32:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.637604087s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................XXX.................] duration: 2.436265367s FAIL (2025-08-17 04:32:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Aug 17 04:21:58.824378996 (202h56m20.783476241s) [upload=211227]: Starting upload Aug 17 04:21:58.824453728 (202h56m20.783550973s) [upload=211227]: Done starting upload Aug 17 04:21:58.824533712 (202h56m20.783630943s) [upload=211227]: Uploading segments... Aug 17 04:21:58.824814622 (202h56m20.783911863s) [upload=211227]: Got next segment. Inline: false Aug 17 04:21:58.824870724 (202h56m20.78396797s) [upload=211227]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 17 04:21:58.824876461 (202h56m20.783973706s) [upload=211227]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 17 04:21:59.025160521 (202h56m20.984257797s) [seg_pos={0 0}, upload=211227]: Begin upload segment... Aug 17 04:21:59.025238019 (202h56m20.984335266s) [seg_pos={0 0}, upload=211227]: Done begin upload segment. Aug 17 04:21:59.025273686 (202h56m20.98437092s) [noise=true , address=64.49.13.26:4951 , piece_id=SJYX5R2QSB64W6M4H5XJCPGSNF5YMJYBRFJF7FPGSZQB4KBAFO6A, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025331026 (202h56m20.984428309s) [noise=true , address=79.127.201.215:4357 , piece_id=5GN73I7DFN6XLU3D7VWZ7TTLYN2PQ2OAUVRRN4YOCZUUWL347EPQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025756251 (202h56m20.984853496s) [noise=true , address=45.91.116.12:4256 , piece_id=OITDESJQZQDVMAEJBQU4KAXWBY43JKXR7AHSE7OQ2TQKG5UKMQPQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.025907425 (202h56m20.98500468s) [noise=true , address=104.36.82.35:4603 , piece_id=ZFG3KRVDGFTJPRYVOB7YFHAJDVS3FREBSOTHPZNHIAOOKNMAPJTA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026127117 (202h56m20.98522437s) [noise=true , address=77.90.31.14:4673 , piece_id=GANKYC2PWGNMZXLY7ZTB5RUFQLTFDZW3CH4AXQJ4FYD673I4WTWQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026350346 (202h56m20.985447594s) [noise=true , address=104.36.82.44:4901 , piece_id=QYKMZVHQ3PNWZ2NPWR4QQNDJWFPZW3PKUQIP6BA2YAQA55R6JWDA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026436670 (202h56m20.985533916s) [noise=true , address=77.90.31.19:4844 , piece_id=JKFXI6B5LZIIXLMF2UN3N226X3YX5ZRWPDJI6DYGRVZRAEOZEEIA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026714369 (202h56m20.985811614s) [noise=true , address=77.90.31.11:4580 , piece_id=TS52Z2GSFQMC722KXJCVYXNMW7UT73PGVWMQFBAXSJZE77CTP7UA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.026788896 (202h56m20.985886144s) [noise=true , address=77.90.31.13:4641 , piece_id=SA6Z2Y3GPMFS2KSQNG2OX5QTEQ6UQMCSI7SAY2BR63NNOS37OTAA, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027068637 (202h56m20.986165882s) [noise=true , address=79.127.220.101:4068 , piece_id=WHJJIXY6ZBZJNQQKJEML5XP5DMRJGUTDRFMOWWD6DJNDSUWCEI6A, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027120600 (202h56m20.986217846s) [noise=true , address=77.90.31.16:4745 , piece_id=SSPWQSSSZNDOSAVYSFO3WKTOH3MZT5WRW5S3ACYWYQGX54R7BDKQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027420110 (202h56m20.986517355s) [noise=true , address=104.36.82.60:4365 , piece_id=4BYPCTHJK6JQKEDQFTQJD6CYZQFO2GGB4WSX7GWTRC4BT6QS5JEQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027425372 (202h56m20.98652262s) [noise=true , address=77.90.31.10:4524 , piece_id=7KW7HUNUBCJWMG66YHQJ63H2UERBGQTIUV24U4WARUQWMHXEFSQQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.027839122 (202h56m20.986936376s) [noise=true , address=66.23.193.175:4553 , piece_id=DQWBH4FPAYOHWQKATN7XPI3B5RFBZ4MF7WUMVS6FR7L5CXNW3JCQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.028247142 (202h56m20.987344396s) [noise=true , address=104.36.82.5:4866 , piece_id=D54LIPVYHKYLIVZRYBOMNOLVYSK3EDH7WDEZ5E563AQ35FUBELAQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.030005732 (202h56m20.989102986s) [noise=true , address=79.127.205.246:4275 , piece_id=ANNA5TM2H5ATR4ZCFZUW5Z2EFFKD6U2X2BWE7MB43RMPE4I7TADQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.030432900 (202h56m20.989530147s) [noise=true , address=45.91.116.11:4132 , piece_id=YPR6JXF4CAZBHP5XZUA5QIK2RFG75HHIRYSQ33EDTNSRT545YY7Q, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.031846491 (202h56m20.990943746s) [noise=true , address=104.36.82.23:4447 , piece_id=A56LTPVK6MNLOUCFJ7FXSAWFUOVQRLMUJB4PS5CYIEQTIWIG7DKQ, seg_pos={0 0}, upload=211227]: Uploading piece... Aug 17 04:21:59.588920323 (202h56m21.548017569s) [upload=211227]: Next returned nil segment Aug 17 04:21:59.588944741 (202h56m21.548041972s) [upload=211227]: Waiting for error group managing segments... Aug 17 04:21:59.741262552 (202h56m21.700359791s) [noise=true , address=45.91.116.12:4256 , piece_id=OITDESJQZQDVMAEJBQU4KAXWBY43JKXR7AHSE7OQ2TQKG5UKMQPQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.742768290 (202h56m21.701865528s) [noise=true , address=77.90.31.11:4580 , piece_id=TS52Z2GSFQMC722KXJCVYXNMW7UT73PGVWMQFBAXSJZE77CTP7UA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.777622259 (202h56m21.736719503s) [noise=true , address=77.90.31.14:4673 , piece_id=GANKYC2PWGNMZXLY7ZTB5RUFQLTFDZW3CH4AXQJ4FYD673I4WTWQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.808692811 (202h56m21.767790057s) [noise=true , address=77.90.31.16:4745 , piece_id=SSPWQSSSZNDOSAVYSFO3WKTOH3MZT5WRW5S3ACYWYQGX54R7BDKQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.831688332 (202h56m21.790785579s) [noise=true , address=77.90.31.19:4844 , piece_id=JKFXI6B5LZIIXLMF2UN3N226X3YX5ZRWPDJI6DYGRVZRAEOZEEIA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.871745680 (202h56m21.830842918s) [noise=true , address=77.90.31.10:4524 , piece_id=7KW7HUNUBCJWMG66YHQJ63H2UERBGQTIUV24U4WARUQWMHXEFSQQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.883314367 (202h56m21.84241162s) [noise=true , address=77.90.31.13:4641 , piece_id=SA6Z2Y3GPMFS2KSQNG2OX5QTEQ6UQMCSI7SAY2BR63NNOS37OTAA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.890311438 (202h56m21.849408688s) [noise=true , address=79.127.220.101:4068 , piece_id=WHJJIXY6ZBZJNQQKJEML5XP5DMRJGUTDRFMOWWD6DJNDSUWCEI6A, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.940441979 (202h56m21.899539228s) [noise=true , address=45.91.116.11:4132 , piece_id=YPR6JXF4CAZBHP5XZUA5QIK2RFG75HHIRYSQ33EDTNSRT545YY7Q, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:21:59.976656973 (202h56m21.935754218s) [noise=true , address=79.127.201.215:4357 , piece_id=5GN73I7DFN6XLU3D7VWZ7TTLYN2PQ2OAUVRRN4YOCZUUWL347EPQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.283185203 (202h56m22.242282448s) [noise=true , address=66.23.193.175:4553 , piece_id=DQWBH4FPAYOHWQKATN7XPI3B5RFBZ4MF7WUMVS6FR7L5CXNW3JCQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.284822985 (202h56m22.243920238s) [noise=true , address=64.49.13.26:4951 , piece_id=SJYX5R2QSB64W6M4H5XJCPGSNF5YMJYBRFJF7FPGSZQB4KBAFO6A, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:00.446988508 (202h56m22.406085753s) [noise=true , address=79.127.205.246:4275 , piece_id=ANNA5TM2H5ATR4ZCFZUW5Z2EFFKD6U2X2BWE7MB43RMPE4I7TADQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: <nil> Aug 17 04:22:58.545488378 (202h57m20.504585625s) [noise=true , address=104.36.82.5:4866 , piece_id=D54LIPVYHKYLIVZRYBOMNOLVYSK3EDH7WDEZ5E563AQ35FUBELAQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:104.36.82.5:4866): protocol: write tcp 139.59.144.22:57798->104.36.82.5:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545674721 (202h57m20.504771972s) [noise=true , address=104.36.82.23:4447 , piece_id=A56LTPVK6MNLOUCFJ7FXSAWFUOVQRLMUJB4PS5CYIEQTIWIG7DKQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, address:104.36.82.23:4447): protocol: write tcp 139.59.144.22:57190->104.36.82.23:4447: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545824338 (202h57m20.504921585s) [noise=true , address=104.36.82.44:4901 , piece_id=QYKMZVHQ3PNWZ2NPWR4QQNDJWFPZW3PKUQIP6BA2YAQA55R6JWDA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:104.36.82.44:4901): protocol: write tcp 139.59.144.22:54098->104.36.82.44:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545853626 (202h57m20.504950858s) [noise=true , address=104.36.82.35:4603 , piece_id=ZFG3KRVDGFTJPRYVOB7YFHAJDVS3FREBSOTHPZNHIAOOKNMAPJTA, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:12ASAeTMBu4sY1GuZiSuwTXmhPksA1Te2BVZdcoE9QvpLAmcQX3, address:104.36.82.35:4603): protocol: write tcp 139.59.144.22:49164->104.36.82.35:4603: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545907146 (202h57m20.505004393s) [noise=true , address=104.36.82.60:4365 , piece_id=4BYPCTHJK6JQKEDQFTQJD6CYZQFO2GGB4WSX7GWTRC4BT6QS5JEQ, seg_pos={0 0}, upload=211227]: Done uploading piece. err: ecclient: upload failed (node:122EX4PNuiRTCG2gbgbaL5dAtD8s5SN7ZesShk7NqUTGCZ3gA2a, address:104.36.82.60:4365): protocol: write tcp 139.59.144.22:43990->104.36.82.60:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:22:58.545989950 (202h57m20.505087195s) [seg_pos={0 0}, upload=211227]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 17 04:22:58.646357515 (202h57m20.605454767s) [upload=211227]: Done uploading segments... Aug 17 04:22:58.646391150 (202h57m20.605488398s) [upload=211227]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.436509317s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XXXX.............] duration: 2.628423385s FAIL (2025-08-17 04:32:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277217340 (202h56m58.236314588s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277444018 (202h56m58.236541264s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277693944 (202h56m58.236791191s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.277957287 (202h56m58.237054531s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278192843 (202h56m58.237290088s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278363507 (202h56m58.237460782s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.278887870 (202h56m58.237985121s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279265735 (202h56m58.238362991s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279639846 (202h56m58.238737091s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.279971414 (202h56m58.239068662s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280041108 (202h56m58.239138356s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280352792 (202h56m58.239450046s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280683314 (202h56m58.239780567s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.280969649 (202h56m58.240066896s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281241193 (202h56m58.240338438s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.281508639 (202h56m58.240605887s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Uploading piece... Aug 17 04:22:36.837376257 (202h56m58.796473489s) [upload=211240]: Next returned nil segment Aug 17 04:22:36.837399552 (202h56m58.796496798s) [upload=211240]: Waiting for error group managing segments... Aug 17 04:22:36.989047654 (202h56m58.948144887s) [noise=true , address=77.90.31.16:4761 , piece_id=BBQ6MRRN7VFPW4TERHX4OBGS5TABKCBX7E35MMXKNTROK2HCFYWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991378317 (202h56m58.950475552s) [noise=true , address=45.91.116.11:4125 , piece_id=7PHLM54VFQQWBFHCUCHXETPAJWQEKVV364UA2X26XMJGNTNL7AWQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:36.991998552 (202h56m58.951095785s) [noise=true , address=77.90.31.18:4823 , piece_id=X45AK2ZYZ5CO3EFPMS26ZACAVKQ7G6G22VG2QLI6XPB6P5IR7WYQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.010955158 (202h56m58.970052395s) [noise=true , address=77.90.31.19:4869 , piece_id=3QEFUP2SUEHZLB2PTCKJTLTEAOUZ3U4GU4K2NY5W7TIXCGVCWGVA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.017235751 (202h56m58.97633299s) [noise=true , address=77.90.31.21:4903 , piece_id=B2O6UPVOQEHVLGUXLAXAOE2GCGB4CALQZIYSWBHEYNEVHQWNAK3A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.045617264 (202h56m59.004714496s) [noise=true , address=77.90.31.10:4538 , piece_id=TXOA5355CRWXU2L2GL2EABPO2L3DR5BAQFUZVFQCZVQRBEZP66OA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.080535376 (202h56m59.039632621s) [noise=true , address=77.90.31.15:4701 , piece_id=IPGE5OY27GVJD65STT7JXTQCN7HSEOQAKNWLF4BIL66SCMIMFIHQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.121242355 (202h56m59.080339591s) [noise=true , address=45.91.116.12:4230 , piece_id=5Z44YMK6XO2UGZOH55FQDOMMAV3QPZH2XMW2RJHWSD6YEOYPM72Q, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.296208684 (202h56m59.255305934s) [noise=true , address=156.146.43.225:4453 , piece_id=D43T2X6NT2R5CU4ACAJJXKI7XMDXEV7GWV4KQ5TZCDO6BOC3JPIA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:22:37.528205336 (202h56m59.487302581s) [noise=true , address=66.23.193.175:4557 , piece_id=TCM2G7Z7J6R7SJAJWSU7PIJEPRQB6RYRTYMIXJVPFMYSHVPAH3IA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: <nil> Aug 17 04:23:35.761418960 (202h57m57.720516213s) [noise=true , address=104.36.82.60:4341 , piece_id=MKJHIPSBIZK7U5BL225UJVURW6L3QK4NFXLGZFJOH2LP3TM6U2SA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1okrDyM4TfzYN3p8zUjTqbw2661rKQQqZdP1ptAKzzTQ82C7cN, address:104.36.82.60:4341): protocol: rpc: dial tcp 104.36.82.60:4341: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761509741 (202h57m57.720606987s) [noise=true , address=104.36.82.34:4556 , piece_id=M4IYC36BRDPKHAQL4BRBM7U7IN4YQLRG5RRXVFXZ535NO5AGAQMA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1i62GRFASf7UPVGhUofmRkz3zZSyhq4fDycgyYtfD7pHR3w91v, address:104.36.82.34:4556): protocol: write tcp 139.59.144.22:37150->104.36.82.34:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761574924 (202h57m57.720672174s) [noise=true , address=104.36.82.17:4333 , piece_id=CGHZEP2ZKWES3L6XH4LSFLOFKYLQPYJZ56G36TSWA2MYHJORAL5A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1xRdTZoEk3NsnS445GcdEvwAdDzNLwUYSfgjB7bBSGRx9L9mrU, address:104.36.82.17:4333): protocol: write tcp 139.59.144.22:47980->104.36.82.17:4333: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761620359 (202h57m57.720717605s) [noise=true , address=104.36.82.58:4293 , piece_id=IY3EUM3SZ42OZA42BC2FSVTLTHU5POSMOOQT6MTLHNPZ6FEYE2EQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:15i783VxxzbriD7LZCbFWNBY8uCKUffXtmWE2eu2VupgE1t9bZ, address:104.36.82.58:4293): protocol: write tcp 139.59.144.22:39172->104.36.82.58:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761661063 (202h57m57.720758308s) [noise=true , address=104.36.82.36:4630 , piece_id=52F2JKA67HIID76G4BPIWQ7FIRZIDYLB5R6KXA2NZC45ULKY4ZXA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:12aETKgTzKw8fyYqbeDA2JzfxZnWn12xUPYK1eRt6TvBPPdRJDp, address:104.36.82.36:4630): protocol: write tcp 139.59.144.22:51970->104.36.82.36:4630: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761681516 (202h57m57.720778766s) [noise=true , address=104.36.82.23:4474 , piece_id=2TJQ3BUKL5BTVHPAOEPKGQIZS4IX54HPYKIRLRAXLUTLMKXST6RA, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1Fpshye6yGAo3yf71eVpxrCdvoRjpkGX9WEX8KCWYbvuhy5i9j, address:104.36.82.23:4474): protocol: write tcp 139.59.144.22:35702->104.36.82.23:4474: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761728458 (202h57m57.720825693s) [noise=true , address=104.36.82.9:4025 , piece_id=EJABFHKRXGUJKB565A56GVUO3I2ERZZEY4T2PZJ73V6MYF23GL2A, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1244zx11aa7W2RGE7w1qeLfsHLxQ6f8tLJeNcxssa68QywFbX1R, address:104.36.82.9:4025): protocol: write tcp 139.59.144.22:48846->104.36.82.9:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761769868 (202h57m57.720867115s) [noise=true , address=104.36.82.8:4992 , piece_id=NZP6GIHTMJJRNUHGZA4JPKFDEJ6LXESXVP33ZFLGR4QLDPVC7MIQ, seg_pos={0 0}, upload=211240]: Done uploading piece. err: ecclient: upload failed (node:1hbt8HKTunpPKyAzH4XBwGQqTKzcgqPpoWwKZm9JMtaK8P7aVp, address:104.36.82.8:4992): protocol: write tcp 139.59.144.22:40976->104.36.82.8:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 17 04:23:35.761795616 (202h57m57.720892851s) [seg_pos={0 0}, upload=211240]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Aug 17 04:23:35.867721911 (202h57m57.826819144s) [upload=211240]: Done uploading segments... Aug 17 04:23:35.867749737 (202h57m57.826846969s) [upload=211240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 2.628484216s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)