Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X......................X.................................] duration: 6.365349952s FAIL (2025-08-16 16:20:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 6.365396631s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X....X........X......................] duration: 2.886987119s FAIL (2025-08-16 16:19:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.88701346s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 2.493016287s FAIL (2025-08-16 16:19:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.493065277s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....................X..............X........X..............] duration: 2.811782371s FAIL (2025-08-16 16:19:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 2.811828975s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X...........................X.............................] duration: 2.480732026s FAIL (2025-08-16 16:19:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.480767027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X......................X................................] duration: 8.649266729s FAIL (2025-08-16 16:19:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 8.649298305s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......................X..............X........X.............] duration: 3.329449951s FAIL (2025-08-16 16:19:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 3.329489598s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X....X........X.....................] duration: 2.765364752s FAIL (2025-08-16 16:18:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.765391695s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 2.394335819s FAIL (2025-08-16 16:18:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.394386202s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X...........................X............................] duration: 2.637630035s FAIL (2025-08-16 16:18:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.637661964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X......................X...............................] duration: 11.663033104s FAIL (2025-08-16 16:18:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 11.663062496s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................X..............X........X............] duration: 6.755342656s FAIL (2025-08-16 16:18:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 6.755374813s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...........................X...........................] duration: 2.439724653s FAIL (2025-08-16 16:18:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.439755131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X....X........X....................] duration: 2.754635317s FAIL (2025-08-16 16:18:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.755063036s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [........................X..............X........X...........] duration: 4.562470277s FAIL (2025-08-16 16:17:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 4.562513815s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 2.483515293s FAIL (2025-08-16 16:17:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.483636528s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X....X........X...................] duration: 2.705261187s FAIL (2025-08-16 16:17:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.705288327s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X......................X..............................] duration: 7.405700012s FAIL (2025-08-16 16:17:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 7.405737227s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...........................X..........................] duration: 2.646765788s FAIL (2025-08-16 16:16:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.646802072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........................X..............X........X..........] duration: 6.218762574s FAIL (2025-08-16 16:16:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 6.218801405s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 2.397200714s FAIL (2025-08-16 16:16:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.397224961s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X......................X.............................] duration: 7.09905406s FAIL (2025-08-16 16:16:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 7.099095146s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X....X........X..................] duration: 2.993161821s FAIL (2025-08-16 16:16:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.993211208s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........................X..............X........X.........] duration: 10.325485755s FAIL (2025-08-16 16:16:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 10.325531372s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X......................X............................] duration: 6.325259891s FAIL (2025-08-16 16:16:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 6.325295094s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 2.550933389s FAIL (2025-08-16 16:16:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.550977574s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X...........................X.........................] duration: 2.631707726s FAIL (2025-08-16 16:16:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.631840624s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X....X........X.................] duration: 2.800985551s FAIL (2025-08-16 16:15:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.801035717s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........................X..............X........X........] duration: 8.362601868s FAIL (2025-08-16 16:15:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 8.362635781s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X......................X...........................] duration: 4.463363054s FAIL (2025-08-16 16:15:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 4.463392928s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 2.447943687s FAIL (2025-08-16 16:14:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.448037464s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X...........................X........................] duration: 2.59976211s FAIL (2025-08-16 16:14:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.5997946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 2.509306056s FAIL (2025-08-16 16:14:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.50933513s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [............................X..............X........X.......] duration: 6.641386172s FAIL (2025-08-16 16:14:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 6.641480363s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X....X........X................] duration: 2.811585759s FAIL (2025-08-16 16:14:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.811614557s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X......................X..........................] duration: 7.706207234s FAIL (2025-08-16 16:14:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 7.706249472s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X...........................X.......................] duration: 2.555372377s FAIL (2025-08-16 16:14:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.555414715s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X...........................X......................] duration: 2.857661851s FAIL (2025-08-16 16:13:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.857755025s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X....X........X...............] duration: 2.886043611s FAIL (2025-08-16 16:13:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.886068387s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.................XX........................................] duration: 2.508068052s FAIL (2025-08-16 16:13:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.50817639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............................X..............X........X......] duration: 7.711645002s FAIL (2025-08-16 16:13:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 7.711691148s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X...........................X.....................] duration: 2.561771224s FAIL (2025-08-16 16:13:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.561810323s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X....X........X..............] duration: 2.911505714s FAIL (2025-08-16 16:13:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.911533009s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............................X..............X........X.....] duration: 17.17995261s FAIL (2025-08-16 16:13:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 17.179989245s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X......................X.........................] duration: 7.507336255s FAIL (2025-08-16 16:13:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 16 15:51:35.483740689 (190h25m57.442837944s) [upload=198231]: Starting upload Aug 16 15:51:35.483820672 (190h25m57.442917918s) [upload=198231]: Done starting upload Aug 16 15:51:35.484116990 (190h25m57.443214237s) [upload=198231]: Uploading segments... Aug 16 15:51:35.484273435 (190h25m57.443370673s) [upload=198231]: Got next segment. Inline: false Aug 16 15:51:35.484338991 (190h25m57.443436236s) [upload=198231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:51:35.484345510 (190h25m57.443442756s) [upload=198231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:51:35.576806489 (190h25m57.535903745s) [upload=198231]: Next segment err: metaclient: Too Many Requests Aug 16 15:51:35.576837435 (190h25m57.535934692s) [upload=198231]: Done uploading segments... Aug 16 15:51:35.576857302 (190h25m57.535954553s) [upload=198231]: Upload finished. err: metaclient: Too Many Requests Duration: 7.507373435s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.................XX.......................................] duration: 2.549760154s FAIL (2025-08-16 16:12:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.549793517s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...............................X..............X........X....] duration: 5.639805028s FAIL (2025-08-16 16:12:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after a223fd7873902fc0f0bc0384552d7565454953193efb24c294d227e3924099ad != before 8ce2298182cb18658076ca1a4c4dc2505bd9ea2f7823b0ea7edf47088784a579 ======LOGS====== Aug 16 16:09:28.754001180 (190h43m50.713098435s) [upload=198541]: Starting upload Aug 16 16:09:28.754070180 (190h43m50.713167436s) [upload=198541]: Done starting upload Aug 16 16:09:28.754174729 (190h43m50.713271961s) [upload=198541]: Uploading segments... Aug 16 16:09:28.754241191 (190h43m50.713338473s) [upload=198541]: Got next segment. Inline: true Aug 16 16:09:28.754247252 (190h43m50.7133445s) [upload=198541]: Waiting for error group managing segments... Aug 16 16:09:28.754304007 (190h43m50.713401268s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 16:09:28.754309597 (190h43m50.713406852s) [upload=198541]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 16:09:28.754312764 (190h43m50.71341002s) [upload=198541]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 16:09:32.091417311 (190h43m54.050514578s) [upload=198541]: Done uploading segments... Aug 16 16:09:32.091458184 (190h43m54.05055544s) [upload=198541]: Upload finished. err: <nil> Duration:* 5.639842796s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X....X........X.............] duration: 3.089566886s FAIL (2025-08-16 16:12:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.089654838s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.................XX......................................] duration: 2.460951401s FAIL (2025-08-16 16:12:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.460993033s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X...........................X....................] duration: 2.683711912s FAIL (2025-08-16 16:12:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== d=198311]: Uploading segments... Aug 16 15:56:35.362777127 (190h30m57.321874381s) [upload=198311]: Got next segment. Inline: false Aug 16 15:56:35.362846814 (190h30m57.321944059s) [upload=198311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:56:35.362853276 (190h30m57.321950521s) [upload=198311]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:56:35.569529127 (190h30m57.528626376s) [seg_pos={0 0}, upload=198311]: Begin upload segment... Aug 16 15:56:35.569623973 (190h30m57.528721207s) [seg_pos={0 0}, upload=198311]: Done begin upload segment. Aug 16 15:56:35.569774986 (190h30m57.528872231s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.569834614 (190h30m57.528931862s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.570295871 (190h30m57.529393119s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.574625622 (190h30m57.53372288s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.575615192 (190h30m57.534712449s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.576543108 (190h30m57.535640356s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.577583860 (190h30m57.53668111s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.578469976 (190h30m57.537567236s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.579292848 (190h30m57.5383901s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580060859 (190h30m57.539158105s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.580727828 (190h30m57.539825076s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.581401020 (190h30m57.540498269s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.588266115 (190h30m57.547363368s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.591328325 (190h30m57.550425574s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.594266622 (190h30m57.553363877s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.595887075 (190h30m57.554984331s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.601144711 (190h30m57.560241968s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:35.605767693 (190h30m57.564864945s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Uploading piece... Aug 16 15:56:36.161829798 (190h30m58.120927048s) [upload=198311]: Next returned nil segment Aug 16 15:56:36.161850512 (190h30m58.120947767s) [upload=198311]: Waiting for error group managing segments... Aug 16 15:56:36.286318508 (190h30m58.245415765s) [noise=true , address=79.127.220.101:4071 , piece_id=U7A33HNXCWNZ3JAZOUPBCQEVDNMU4P36EYA5SH5QZ2UUHQOUMS6A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12UW4p7c1bwrzjUQ5bXZuDq3TtK3ra1riJe5DmUmUZiTuMhMVAX, address:79.127.220.101:4071): protocol: storage node overloaded, request limit: 2000 Aug 16 15:56:36.439811359 (190h30m58.398908591s) [noise=true , address=77.90.31.13:4646 , piece_id=WHARBVTSCUO4RVAC6NS46F667EUTGAW5L5YX3F44LS3GY7IP47GQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.465239317 (190h30m58.42433655s) [noise=true , address=77.90.31.21:4898 , piece_id=3AACVDHZU2LK4KOX3FWCGEGSXH4JZYWY5LI2YNM2TLNLTROOYAWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.473335184 (190h30m58.432432417s) [noise=true , address=77.90.31.18:4829 , piece_id=K75KOBXZFREOAURQYLDEC3GREGZLK62GDU7SQAZA7DKROMEO77KQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.765760642 (190h30m58.724857894s) [noise=true , address=45.91.116.6:4985 , piece_id=7BQTF3QZZPEI5TCVL34MFIU22SCTRHRDGJLZIUEDS6MW7Y3ZCQKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.805958855 (190h30m58.76505611s) [noise=true , address=45.91.116.11:4125 , piece_id=XPXILTJKIEV2CQEU46YSRPMSBGRWJ3IMJXWIR75HLEMZN722PRMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:36.888530403 (190h30m58.847627663s) [noise=true , address=79.127.205.245:4223 , piece_id=MZM3XGEJIPOLBXRD63LQVYFHJ4VGSYGVVE6CONOZMQW4LMDL7PLA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:37.979697099 (190h30m59.93879433s) [noise=true , address=45.91.116.12:4241 , piece_id=XB74OS3XJ5MABOIQUURG246IESYAQ6WIT2ZLSD4GKBCRR2PBF25A, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:39.217581121 (190h31m1.176678367s) [noise=true , address=64.49.13.139:4473 , piece_id=BCZUR7WAQ6OD7LJO76BUGPMV5MGJEVCC3TEXZKMKDFXFE7ZIXOWA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:44.111995534 (190h31m6.07109278s) [noise=true , address=104.36.82.59:4316 , piece_id=3JZZCO2MUSF44PGYQSBBCJRMPRCXEG4BV3QAWC2TIJTRVK5TPSRQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:46.643054356 (190h31m8.602151595s) [noise=true , address=66.23.193.175:4558 , piece_id=S3ZLRKHTDH6HIOYEZAJOGZTLCOYEGZABGOJTUNOQGFELO7FGLPKA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:47.260142645 (190h31m9.219239885s) [noise=true , address=104.36.82.38:4695 , piece_id=7S4M4AB4APRVBF5GAEZ2W7266MPZCKZNHY6UCSHY73DZULFAHV5Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:56:50.818645623 (190h31m12.777742862s) [noise=true , address=104.36.82.34:4561 , piece_id=IEHTTCOQADCGOPIWEUSHQCDPCPATSSG6K43I7MGJ6MVRUY2IT43Q, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:13.279901900 (190h31m35.238999145s) [noise=true , address=104.36.82.60:4361 , piece_id=BP3GVQIZ2IKPCYMK2QMVZMQN46KTGUUPXV23PJ4JCBYGFARGC3FQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: <nil> Aug 16 15:57:35.089739837 (190h31m57.048837097s) [noise=true , address=104.36.82.23:4465 , piece_id=HJPFREJ2MI46KU7FBMAM5RLP3R52VNUN7KATX23VL7OZLTXTTZPQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1Mt7mGfsy6qpxmZTiKcFavb3o8zYLJdk3tXDtGReLLyBkgu2jw, address:104.36.82.23:4465): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:57:35.090030960 (190h31m57.049128206s) [noise=true , address=104.36.82.2:4737 , piece_id=BEAJWF3446RTUGUACFD56SL7FQW2YITNXKPU5RAQ2KHDXWDKPTBQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59942->104.36.82.2:4737: use of closed network connection Aug 16 15:57:35.090175340 (190h31m57.049272579s) [noise=true , address=77.90.31.12:4619 , piece_id=7W6OGNJHI7MVB43PKKYZV3LMEVU5KIACH54SCDVAE2R5QCF6KIMQ, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Aug 16 15:57:35.090217180 (190h31m57.049314425s) [noise=true , address=77.90.31.15:4727 , piece_id=C5KX4O3AOR7D5IKO5EAG6TA6UN6ODBR3VWF6JT3G55CWYOI7THSA, seg_pos={0 0}, upload=198311]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; context deadline exceeded Aug 16 15:57:35.090267174 (190h31m57.049364425s) [seg_pos={0 0}, upload=198311]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Aug 16 15:57:35.207292877 (190h31m57.166390118s) [upload=198311]: Done uploading segments... Aug 16 15:57:35.207339136 (190h31m57.166436378s) [upload=198311]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.683760553s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)