Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 3.484426292s FAIL (2025-08-16 16:29:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 3.484458234s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X....X........X...................................] duration: 2.764611944s FAIL (2025-08-16 16:29:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.76470601s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........X..............X........X.........................] duration: 1.673633184s FAIL (2025-08-16 16:28:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 1.673665413s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.414147157s FAIL (2025-08-16 16:28:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.414210015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.727357884s FAIL (2025-08-16 16:28:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.72739087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.50642552s FAIL (2025-08-16 16:28:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.506488322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 3.431964903s FAIL (2025-08-16 16:28:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 3.431998866s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 2.398078496s FAIL (2025-08-16 16:28:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.398207842s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X....X........X..................................] duration: 2.811797808s FAIL (2025-08-16 16:28:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.811817505s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.799916144s FAIL (2025-08-16 16:27:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 2.799946604s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X....X........X.................................] duration: 3.014516232s FAIL (2025-08-16 16:27:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.014538992s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........X..............X........X........................] duration: 2.159382732s FAIL (2025-08-16 16:27:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 2.159415422s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.757877287s FAIL (2025-08-16 16:27:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.757914395s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 2.324813504s FAIL (2025-08-16 16:27:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.324844166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [............X..............X........X.......................] duration: 1.429130732s FAIL (2025-08-16 16:27:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 1.429167501s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X....X........X................................] duration: 2.582919262s FAIL (2025-08-16 16:26:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.583030357s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 3.550943637s FAIL (2025-08-16 16:26:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 3.550979921s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.828806898s FAIL (2025-08-16 16:26:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.828850223s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............X..............X........X......................] duration: 1.537788791s FAIL (2025-08-16 16:26:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 1.537892008s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 2.481810642s FAIL (2025-08-16 16:26:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.48184635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 3.738228396s FAIL (2025-08-16 16:26:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 3.738332806s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X....X........X...............................] duration: 2.647102682s FAIL (2025-08-16 16:26:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.647123561s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............X..............X........X.....................] duration: 1.068862106s FAIL (2025-08-16 16:25:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 1.068906086s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 4.502101462s FAIL (2025-08-16 16:25:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 4.502194186s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.649267755s FAIL (2025-08-16 16:25:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.649394276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 2.673546042s FAIL (2025-08-16 16:25:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.673646309s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X....X........X..............................] duration: 2.71294437s FAIL (2025-08-16 16:24:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.712971864s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 2.60286116s FAIL (2025-08-16 16:24:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.602889797s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 3.005523869s FAIL (2025-08-16 16:24:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.005558514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............X..............X........X....................] duration: 2.51332367s FAIL (2025-08-16 16:24:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 2.51337343s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.454459492s FAIL (2025-08-16 16:24:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 6.45451372s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X....X........X.............................] duration: 2.786292857s FAIL (2025-08-16 16:24:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.786378056s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [................X..............X........X...................] duration: 3.014589964s FAIL (2025-08-16 16:24:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 3.014709408s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 2.586860174s FAIL (2025-08-16 16:23:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.586894153s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 4.518944324s FAIL (2025-08-16 16:23:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 4.518972798s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 3.134268659s FAIL (2025-08-16 16:23:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.134354934s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X....X........X............................] duration: 2.729426451s FAIL (2025-08-16 16:23:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.729460146s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 2.457464163s FAIL (2025-08-16 16:23:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.457502768s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X....X........X...........................] duration: 2.798559715s FAIL (2025-08-16 16:23:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.798593603s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 7.211253933s FAIL (2025-08-16 16:23:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 7.211306496s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.................X..............X........X..................] duration: 6.597909394s FAIL (2025-08-16 16:23:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 6.597965903s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 2.450017961s FAIL (2025-08-16 16:22:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.450131159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 7.520452347s FAIL (2025-08-16 16:22:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 7.520542164s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.805436878s FAIL (2025-08-16 16:22:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.805470178s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..................X..............X........X.................] duration: 2.849413733s FAIL (2025-08-16 16:22:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 2.849446928s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X....X........X..........................] duration: 2.799419427s FAIL (2025-08-16 16:22:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.799446835s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X....X........X.........................] duration: 2.819633494s FAIL (2025-08-16 16:22:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.819719034s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [X......................X....................................] duration: 5.129747661s FAIL (2025-08-16 16:21:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 5.129792869s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X...........................X...............................] duration: 2.578057464s FAIL (2025-08-16 16:21:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.578090534s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X......................X...................................] duration: 9.393979872s FAIL (2025-08-16 16:21:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 9.394014738s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)