Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X.................XX] duration: 30.101832985s FAIL (2025-08-16 15:44:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Aug 16 15:43:37.863204100 (190h17m59.822301346s) [upload=198094]: Starting upload Aug 16 15:43:37.863288635 (190h17m59.822385867s) [upload=198094]: Done starting upload Aug 16 15:43:37.863413110 (190h17m59.822510346s) [upload=198094]: Uploading segments... Aug 16 15:43:37.863597613 (190h17m59.822694857s) [upload=198094]: Got next segment. Inline: false Aug 16 15:43:37.863656350 (190h17m59.822753593s) [upload=198094]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:37.863663718 (190h17m59.822760953s) [upload=198094]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:37.970896691 (190h17m59.929993952s) [seg_pos={0 0}, upload=198094]: Begin upload segment... Aug 16 15:43:37.970967703 (190h17m59.930064938s) [seg_pos={0 0}, upload=198094]: Done begin upload segment. Aug 16 15:43:37.971006383 (190h17m59.930103652s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971026376 (190h17m59.930123622s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971104664 (190h17m59.930201909s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971516424 (190h17m59.930613683s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971535546 (190h17m59.930632791s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971890955 (190h17m59.930988201s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.971922365 (190h17m59.931019619s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972314472 (190h17m59.931411732s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972323384 (190h17m59.931420632s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972706299 (190h17m59.93180354s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.972787765 (190h17m59.931885022s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973401030 (190h17m59.932498285s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.973907413 (190h17m59.933004662s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.974550229 (190h17m59.933647485s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.977671370 (190h17m59.936768608s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:37.981429699 (190h17m59.940526945s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Uploading piece... Aug 16 15:43:38.595447387 (190h18m0.554544631s) [upload=198094]: Next returned nil segment Aug 16 15:43:38.595468502 (190h18m0.554565757s) [upload=198094]: Waiting for error group managing segments... Aug 16 15:43:38.730928124 (190h18m0.69002537s) [noise=true , address=45.91.116.8:4047 , piece_id=BAKANUZ53RNYIEAE7HWH2BJSVFBWMI42QTW5MYTTDQDZO5ZEDVEQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.762614705 (190h18m0.721711941s) [noise=true , address=104.36.82.6:4873 , piece_id=PARUNDF7NSKIEKH63VVBIITCIML473H422FYDUOAWZN3ZBFK3D3Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.764453653 (190h18m0.723550884s) [noise=true , address=104.36.82.15:4257 , piece_id=CHYAPKFRVIA5USFCCPASVOYNRCL64DUCCDZU5IHEX6J3WYO45VTA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.777729636 (190h18m0.736826875s) [noise=true , address=104.36.82.12:4151 , piece_id=7A7F4KWB3JQ3D63SUAMDBZZAFWOVWPDYCPS5LYCCBDPXF2C3TRMA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.798956811 (190h18m0.758054043s) [noise=true , address=104.36.82.7:4909 , piece_id=UQVZ64B3OXD34SNJYN2V47M2Y2VHSUF32ICKN77DKGQRIUKXPFGQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.806273762 (190h18m0.765371s) [noise=true , address=45.91.116.7:4010 , piece_id=HSTMZ5NAALIIN6GUL5LXEVNRN6V6WYXTETDDONB4L2ER3DKOFB6A, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.812038196 (190h18m0.771135434s) [noise=true , address=104.36.82.10:4077 , piece_id=2CKC22PRCLSAEJUFYD6R22NCYTGW5M7MCAYGXPZ4U3SKKWMX4YKQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:38.871231215 (190h18m0.830328459s) [noise=true , address=45.91.116.9:4262 , piece_id=URSVZYVRCTOLWFL5VXBWQKWHE2WYMB6BJJUVWCEDF54SWDBAFFVQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:43:39.245208369 (190h18m1.20430561s) [noise=true , address=45.91.116.14:4189 , piece_id=KJXBZ2KJELZL6SEI5MAZKS6GZRRKPG6Z6KFQVOXTBGYGCU4ZMVQA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:06.317776143 (190h18m28.276873395s) [noise=true , address=77.90.31.18:4807 , piece_id=WW7IHRHCZODIGM2K67RU7DJ77X7P2AP4BDRCY6NYIXYO2FERGKZA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.409706341 (190h18m29.36880358s) [noise=true , address=104.36.82.11:4114 , piece_id=DG3NEQNCBYX5RFTFU7TFAOKLAZ42OOMJZZRJSOE7Q557XFO336MQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: <nil> Aug 16 15:44:07.596550188 (190h18m29.555647444s) [noise=true , address=104.36.82.13:4187 , piece_id=NFLXPHP2M5W6HF5YINXX7OU2MTRDWWLIQL32WN264ZWZDSRZREWQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596650344 (190h18m29.555747599s) [noise=true , address=104.36.82.3:4765 , piece_id=PTPKP5RPKORDWAI22CYFPFADAZTCJE37CH6UQWNTPZVZ2ISR55IQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: expected piece hash; context deadline exceeded; EOF Aug 16 15:44:07.596846409 (190h18m29.555943655s) [noise=true , address=77.90.31.10:4520 , piece_id=IA3I64SZJXW6HUW3PCJGDQ2XL4X4UWAKDXTE2RPPS2H7PQMNYC6Q, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:44470->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:44:07.596865147 (190h18m29.555962384s) [noise=true , address=104.36.82.16:4294 , piece_id=2KKZMJTM6SHCWW6C5D7OUEV2CCIEZHR4ZBDBK2Z7I3BYOLL7U2QQ, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54812->104.36.82.16:4294: use of closed network connection Aug 16 15:44:07.596967816 (190h18m29.556065068s) [noise=true , address=104.36.82.4:4803 , piece_id=DYVLJB3FTNQNRHSHYL7CVYJAHYFT7FZB34BQJRWDQXJJAOT2B6SA, seg_pos={0 0}, upload=198094]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; write tcp 139.59.144.22:58270->104.36.82.4:4803: use of closed network connection; piecestore: piecestore close: context deadline exceeded Aug 16 15:44:07.596994400 (190h18m29.556091658s) [seg_pos={0 0}, upload=198094]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Aug 16 15:44:07.695984788 (190h18m29.655082049s) [upload=198094]: Done uploading segments... Aug 16 15:44:07.696036029 (190h18m29.655133287s) [upload=198094]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.101861054s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.................X] duration: 30.112302876s FAIL (2025-08-16 15:43:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Aug 16 15:43:07.622835035 (190h17m29.581932285s) [upload=198084]: Starting upload Aug 16 15:43:07.622942172 (190h17m29.582039419s) [upload=198084]: Done starting upload Aug 16 15:43:07.623031604 (190h17m29.582128854s) [upload=198084]: Uploading segments... Aug 16 15:43:07.623387386 (190h17m29.582484623s) [upload=198084]: Got next segment. Inline: false Aug 16 15:43:07.623441663 (190h17m29.582538918s) [upload=198084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:43:07.623452950 (190h17m29.582550202s) [upload=198084]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:43:07.734201384 (190h17m29.693298644s) [seg_pos={0 0}, upload=198084]: Begin upload segment... Aug 16 15:43:07.734281124 (190h17m29.693378369s) [seg_pos={0 0}, upload=198084]: Done begin upload segment. Aug 16 15:43:07.734324112 (190h17m29.693421362s) [noise=true , address=45.91.116.7:4011 , piece_id=L52OSPTBUDTGCSUMZIDL2EW54J7AIJYXITXC7QWCHYVN4NYEMT7Q, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.734352807 (190h17m29.693450062s) [noise=true , address=45.91.116.9:4261 , piece_id=4GL2AM3WJCUCTKABDOCKZYRJN32KPYGJPU3NTHOQNOSZ7FGGSWZA, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.734801717 (190h17m29.693898968s) [noise=true , address=104.36.82.6:4876 , piece_id=E2CRSYPOHOVTDGDWFF6T4TZLIL7BNYP6B4XI7CR36IKCVAU2AXZQ, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.734820106 (190h17m29.693917355s) [noise=true , address=45.91.116.5:4297 , piece_id=5632ZHQWXBS66OW4CR5UHEH7DCJOYMRUQS3X37QF7FFOUX3JMJDA, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.735228202 (190h17m29.694325451s) [noise=true , address=77.90.31.14:4663 , piece_id=TEGACRI7HO27B4ZI3QV62N7SIDSEQ4BE4ZQW77MKYV7O5GGKFJPQ, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.735238542 (190h17m29.694335799s) [noise=true , address=104.36.82.4:4804 , piece_id=3QYWY4AWHJX3U4WBOBQ2OKDBJ3KPC4YTWSKR3UJG2H7QKB3GLZYQ, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.735627561 (190h17m29.694724809s) [noise=true , address=104.36.82.12:4149 , piece_id=QY7DO7YY7R3CG5P3P5UKU25SO6EYYIDB47EEN2MBWOKSWGR3NZ6Q, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.735647590 (190h17m29.694744847s) [noise=true , address=104.36.82.2:4731 , piece_id=ZL4OSIWH7TC3OX4JUN6DFOYEKN7QCWJBIJAX4ROLPTDM3SGUDP2A, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.736036851 (190h17m29.695134087s) [noise=true , address=45.91.116.8:4048 , piece_id=J536X5BXF6OERIFMFNYLXR25QHKTWWJ7DPSNSIIP4QXMTYHAUWWA, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.736047754 (190h17m29.695145s) [noise=true , address=77.90.31.11:4555 , piece_id=HS5JACTQ5UUE3CAAXLDUDVM6CNVC76QWUT2VQKWFXOF5EL3O2EQA, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.736425865 (190h17m29.695523115s) [noise=true , address=104.36.82.11:4115 , piece_id=ZZLZE25XOWBWJDDH7FEEYR74YPCVSBVI7CQD6GTG6CQMEFGHK2KA, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.737634028 (190h17m29.696731264s) [noise=true , address=104.36.82.5:4839 , piece_id=P47CLM36SLM6E7UKL5YYRVUCTYMMGE63PHFN6QNQR4GVT325GR5A, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.738181454 (190h17m29.697278704s) [noise=true , address=104.36.82.7:4912 , piece_id=HQSFCGME3FF7TUPOXD4GUJUQEF2Z5YWAZQGOAAZKILZBCGUBCW7Q, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.738601775 (190h17m29.697699024s) [noise=true , address=104.36.82.8:4971 , piece_id=WR6X7OIPS2522GSM6L6JG4OJRAOD5GMEETCJOFHEQCRKZZQROTCQ, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.738997566 (190h17m29.698094821s) [noise=true , address=45.91.116.6:4974 , piece_id=ZEDZDURBN632VT57V5OQX5HNQE4EPEQPHW5PVRDO535EUDXPXTCQ, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:07.740149568 (190h17m29.699246809s) [noise=true , address=77.90.31.18:4807 , piece_id=5HZXGFQK6ZMM6NW6SOT7UKVR5KWA2RKNPJYHG7WYBOK6KZ57JRIA, seg_pos={0 0}, upload=198084]: Uploading piece... Aug 16 15:43:08.350731304 (190h17m30.30982855s) [upload=198084]: Next returned nil segment Aug 16 15:43:08.350750069 (190h17m30.309847317s) [upload=198084]: Waiting for error group managing segments... Aug 16 15:43:08.523524057 (190h17m30.482621306s) [noise=true , address=104.36.82.6:4876 , piece_id=E2CRSYPOHOVTDGDWFF6T4TZLIL7BNYP6B4XI7CR36IKCVAU2AXZQ, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.530214696 (190h17m30.489311951s) [noise=true , address=77.90.31.11:4555 , piece_id=HS5JACTQ5UUE3CAAXLDUDVM6CNVC76QWUT2VQKWFXOF5EL3O2EQA, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.538823924 (190h17m30.497921172s) [noise=true , address=104.36.82.7:4912 , piece_id=HQSFCGME3FF7TUPOXD4GUJUQEF2Z5YWAZQGOAAZKILZBCGUBCW7Q, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.587643867 (190h17m30.546741114s) [noise=true , address=104.36.82.12:4149 , piece_id=QY7DO7YY7R3CG5P3P5UKU25SO6EYYIDB47EEN2MBWOKSWGR3NZ6Q, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.591865251 (190h17m30.550962501s) [noise=true , address=45.91.116.7:4011 , piece_id=L52OSPTBUDTGCSUMZIDL2EW54J7AIJYXITXC7QWCHYVN4NYEMT7Q, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.602422635 (190h17m30.561519889s) [noise=true , address=77.90.31.14:4663 , piece_id=TEGACRI7HO27B4ZI3QV62N7SIDSEQ4BE4ZQW77MKYV7O5GGKFJPQ, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.613558414 (190h17m30.572655661s) [noise=true , address=45.91.116.9:4261 , piece_id=4GL2AM3WJCUCTKABDOCKZYRJN32KPYGJPU3NTHOQNOSZ7FGGSWZA, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.625333870 (190h17m30.584431127s) [noise=true , address=45.91.116.6:4974 , piece_id=ZEDZDURBN632VT57V5OQX5HNQE4EPEQPHW5PVRDO535EUDXPXTCQ, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.632179694 (190h17m30.591276939s) [noise=true , address=45.91.116.5:4297 , piece_id=5632ZHQWXBS66OW4CR5UHEH7DCJOYMRUQS3X37QF7FFOUX3JMJDA, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:08.887245127 (190h17m30.846342377s) [noise=true , address=45.91.116.8:4048 , piece_id=J536X5BXF6OERIFMFNYLXR25QHKTWWJ7DPSNSIIP4QXMTYHAUWWA, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:25.821185212 (190h17m47.780282468s) [noise=true , address=104.36.82.11:4115 , piece_id=ZZLZE25XOWBWJDDH7FEEYR74YPCVSBVI7CQD6GTG6CQMEFGHK2KA, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:36.923306399 (190h17m58.882403632s) [noise=true , address=104.36.82.4:4804 , piece_id=3QYWY4AWHJX3U4WBOBQ2OKDBJ3KPC4YTWSKR3UJG2H7QKB3GLZYQ, seg_pos={0 0}, upload=198084]: Done uploading piece. err: <nil> Aug 16 15:43:37.351811885 (190h17m59.31090912s) [noise=true , address=104.36.82.2:4731 , piece_id=ZL4OSIWH7TC3OX4JUN6DFOYEKN7QCWJBIJAX4ROLPTDM3SGUDP2A, seg_pos={0 0}, upload=198084]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 139.59.144.22:53300->104.36.82.2:4731: use of closed network connection; write tcp 139.59.144.22:53300->104.36.82.2:4731: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53300->104.36.82.2:4731: use of closed network connection Aug 16 15:43:37.351930005 (190h17m59.311027244s) [noise=true , address=104.36.82.5:4839 , piece_id=P47CLM36SLM6E7UKL5YYRVUCTYMMGE63PHFN6QNQR4GVT325GR5A, seg_pos={0 0}, upload=198084]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 139.59.144.22:54380->104.36.82.5:4839: use of closed network connection; write tcp 139.59.144.22:54380->104.36.82.5:4839: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:54380->104.36.82.5:4839: use of closed network connection Aug 16 15:43:37.352017308 (190h17m59.311114553s) [noise=true , address=77.90.31.18:4807 , piece_id=5HZXGFQK6ZMM6NW6SOT7UKVR5KWA2RKNPJYHG7WYBOK6KZ57JRIA, seg_pos={0 0}, upload=198084]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 139.59.144.22:41680->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:43:37.352103504 (190h17m59.311200756s) [noise=true , address=104.36.82.8:4971 , piece_id=WR6X7OIPS2522GSM6L6JG4OJRAOD5GMEETCJOFHEQCRKZZQROTCQ, seg_pos={0 0}, upload=198084]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:43838->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:43838->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43838->104.36.82.8:4971: use of closed network connection Aug 16 15:43:37.352146090 (190h17m59.311243347s) [seg_pos={0 0}, upload=198084]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Aug 16 15:43:37.460521948 (190h17m59.419619203s) [upload=198084]: Done uploading segments... Aug 16 15:43:37.460555789 (190h17m59.419653042s) [upload=198084]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.112354372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XX...X..........................X............] duration: 38.022594213s FAIL (2025-08-16 15:43:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 38.022625185s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...........X....................] duration: 2.29859031s FAIL (2025-08-16 15:43:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 2.298635855s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 20.899064674s FAIL (2025-08-16 15:43:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 20.899113831s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 20.997163459s FAIL (2025-08-16 15:42:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:28:46.883135980 (190h3m8.842233233s) [upload=197828]: Starting upload Aug 16 15:28:46.883238479 (190h3m8.842335732s) [upload=197828]: Done starting upload Aug 16 15:28:46.883377123 (190h3m8.842474378s) [upload=197828]: Uploading segments... Aug 16 15:28:46.883631985 (190h3m8.842729235s) [upload=197828]: Got next segment. Inline: false Aug 16 15:28:46.883695796 (190h3m8.842793047s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:46.883703737 (190h3m8.842800987s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:28:47.148254023 (190h3m9.10735128s) [seg_pos={0 0}, upload=197828]: Begin upload segment... Aug 16 15:28:47.148325960 (190h3m9.107423206s) [seg_pos={0 0}, upload=197828]: Done begin upload segment. Aug 16 15:28:47.148366495 (190h3m9.107463745s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149105833 (190h3m9.108203083s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149659987 (190h3m9.108757238s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150048977 (190h3m9.109146233s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150192097 (190h3m9.109289346s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150487633 (190h3m9.109584885s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150890236 (190h3m9.109987491s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151281889 (190h3m9.11037914s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151630546 (190h3m9.110727799s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151973595 (190h3m9.11107085s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152305026 (190h3m9.111402283s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152634492 (190h3m9.111731742s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.153725357 (190h3m9.112822608s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.154890697 (190h3m9.113987948s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155414981 (190h3m9.114512226s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155964934 (190h3m9.115062197s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.789275832 (190h3m9.748373079s) [upload=197828]: Next returned nil segment Aug 16 15:28:47.789293260 (190h3m9.748390507s) [upload=197828]: Waiting for error group managing segments... Aug 16 15:28:47.942915173 (190h3m9.902012419s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:47.948000865 (190h3m9.907098103s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:52.322732270 (190h3m14.28182951s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:54.226556301 (190h3m16.185653539s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.082436730 (190h3m22.041533992s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.359706605 (190h3m22.318803836s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.641440409 (190h3m22.600537655s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.795956621 (190h3m22.75505386s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:02.308650030 (190h3m24.267747277s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.278922536 (190h3m31.238019793s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.328348624 (190h3m31.287445869s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:10.659976400 (190h3m32.619073639s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983246801 (190h3m35.942344047s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983274710 (190h3m35.942371956s) [seg_pos={0 0}, upload=197828]: Segment reached optimal threshold of 13 pieces. Aug 16 15:29:13.983464694 (190h3m35.942561926s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983650839 (190h3m35.942748071s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection Aug 16 15:29:13.983715300 (190h3m35.942812531s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983728315 (190h3m35.942825563s) [seg_pos={0 0}, upload=197828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:29:13.983839210 (190h3m35.942936459s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:29:13.983862244 (190h3m35.94295949s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:29:14.175384438 (190h3m36.134481686s) [upload=197828]: Done uploading segments... Aug 16 15:29:14.175404292 (190h3m36.134501536s) [upload=197828]: Upload finished. err: <nil> Duration: 20.997244154s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 21.942834396s FAIL (2025-08-16 15:42:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 21.94285953s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 19.335585456s FAIL (2025-08-16 15:42:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:28:46.883135980 (190h3m8.842233233s) [upload=197828]: Starting upload Aug 16 15:28:46.883238479 (190h3m8.842335732s) [upload=197828]: Done starting upload Aug 16 15:28:46.883377123 (190h3m8.842474378s) [upload=197828]: Uploading segments... Aug 16 15:28:46.883631985 (190h3m8.842729235s) [upload=197828]: Got next segment. Inline: false Aug 16 15:28:46.883695796 (190h3m8.842793047s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:46.883703737 (190h3m8.842800987s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:28:47.148254023 (190h3m9.10735128s) [seg_pos={0 0}, upload=197828]: Begin upload segment... Aug 16 15:28:47.148325960 (190h3m9.107423206s) [seg_pos={0 0}, upload=197828]: Done begin upload segment. Aug 16 15:28:47.148366495 (190h3m9.107463745s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149105833 (190h3m9.108203083s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149659987 (190h3m9.108757238s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150048977 (190h3m9.109146233s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150192097 (190h3m9.109289346s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150487633 (190h3m9.109584885s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150890236 (190h3m9.109987491s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151281889 (190h3m9.11037914s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151630546 (190h3m9.110727799s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151973595 (190h3m9.11107085s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152305026 (190h3m9.111402283s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152634492 (190h3m9.111731742s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.153725357 (190h3m9.112822608s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.154890697 (190h3m9.113987948s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155414981 (190h3m9.114512226s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155964934 (190h3m9.115062197s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.789275832 (190h3m9.748373079s) [upload=197828]: Next returned nil segment Aug 16 15:28:47.789293260 (190h3m9.748390507s) [upload=197828]: Waiting for error group managing segments... Aug 16 15:28:47.942915173 (190h3m9.902012419s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:47.948000865 (190h3m9.907098103s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:52.322732270 (190h3m14.28182951s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:54.226556301 (190h3m16.185653539s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.082436730 (190h3m22.041533992s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.359706605 (190h3m22.318803836s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.641440409 (190h3m22.600537655s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.795956621 (190h3m22.75505386s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:02.308650030 (190h3m24.267747277s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.278922536 (190h3m31.238019793s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.328348624 (190h3m31.287445869s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:10.659976400 (190h3m32.619073639s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983246801 (190h3m35.942344047s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983274710 (190h3m35.942371956s) [seg_pos={0 0}, upload=197828]: Segment reached optimal threshold of 13 pieces. Aug 16 15:29:13.983464694 (190h3m35.942561926s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983650839 (190h3m35.942748071s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection Aug 16 15:29:13.983715300 (190h3m35.942812531s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983728315 (190h3m35.942825563s) [seg_pos={0 0}, upload=197828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:29:13.983839210 (190h3m35.942936459s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:29:13.983862244 (190h3m35.94295949s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:29:14.175384438 (190h3m36.134481686s) [upload=197828]: Done uploading segments... Aug 16 15:29:14.175404292 (190h3m36.134501536s) [upload=197828]: Upload finished. err: <nil> Duration: 19.335612269s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [................XX...X..........................X...........] duration: 20.352681436s FAIL (2025-08-16 15:42:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 20.35271411s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 30.437219416s FAIL (2025-08-16 15:42:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 30.437272106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 29.583744982s FAIL (2025-08-16 15:42:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:28:46.883135980 (190h3m8.842233233s) [upload=197828]: Starting upload Aug 16 15:28:46.883238479 (190h3m8.842335732s) [upload=197828]: Done starting upload Aug 16 15:28:46.883377123 (190h3m8.842474378s) [upload=197828]: Uploading segments... Aug 16 15:28:46.883631985 (190h3m8.842729235s) [upload=197828]: Got next segment. Inline: false Aug 16 15:28:46.883695796 (190h3m8.842793047s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:46.883703737 (190h3m8.842800987s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:28:47.148254023 (190h3m9.10735128s) [seg_pos={0 0}, upload=197828]: Begin upload segment... Aug 16 15:28:47.148325960 (190h3m9.107423206s) [seg_pos={0 0}, upload=197828]: Done begin upload segment. Aug 16 15:28:47.148366495 (190h3m9.107463745s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149105833 (190h3m9.108203083s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149659987 (190h3m9.108757238s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150048977 (190h3m9.109146233s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150192097 (190h3m9.109289346s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150487633 (190h3m9.109584885s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150890236 (190h3m9.109987491s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151281889 (190h3m9.11037914s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151630546 (190h3m9.110727799s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151973595 (190h3m9.11107085s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152305026 (190h3m9.111402283s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152634492 (190h3m9.111731742s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.153725357 (190h3m9.112822608s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.154890697 (190h3m9.113987948s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155414981 (190h3m9.114512226s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155964934 (190h3m9.115062197s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.789275832 (190h3m9.748373079s) [upload=197828]: Next returned nil segment Aug 16 15:28:47.789293260 (190h3m9.748390507s) [upload=197828]: Waiting for error group managing segments... Aug 16 15:28:47.942915173 (190h3m9.902012419s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:47.948000865 (190h3m9.907098103s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:52.322732270 (190h3m14.28182951s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:54.226556301 (190h3m16.185653539s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.082436730 (190h3m22.041533992s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.359706605 (190h3m22.318803836s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.641440409 (190h3m22.600537655s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.795956621 (190h3m22.75505386s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:02.308650030 (190h3m24.267747277s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.278922536 (190h3m31.238019793s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.328348624 (190h3m31.287445869s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:10.659976400 (190h3m32.619073639s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983246801 (190h3m35.942344047s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983274710 (190h3m35.942371956s) [seg_pos={0 0}, upload=197828]: Segment reached optimal threshold of 13 pieces. Aug 16 15:29:13.983464694 (190h3m35.942561926s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983650839 (190h3m35.942748071s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection Aug 16 15:29:13.983715300 (190h3m35.942812531s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983728315 (190h3m35.942825563s) [seg_pos={0 0}, upload=197828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:29:13.983839210 (190h3m35.942936459s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:29:13.983862244 (190h3m35.94295949s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:29:14.175384438 (190h3m36.134481686s) [upload=197828]: Done uploading segments... Aug 16 15:29:14.175404292 (190h3m36.134501536s) [upload=197828]: Upload finished. err: <nil> Duration: 29.583846966s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............................X...........X...................] duration: 1.729011097s FAIL (2025-08-16 15:42:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.72906802s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 27.888337822s FAIL (2025-08-16 15:41:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 27.888387911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 14.078320704s FAIL (2025-08-16 15:41:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:28:46.883135980 (190h3m8.842233233s) [upload=197828]: Starting upload Aug 16 15:28:46.883238479 (190h3m8.842335732s) [upload=197828]: Done starting upload Aug 16 15:28:46.883377123 (190h3m8.842474378s) [upload=197828]: Uploading segments... Aug 16 15:28:46.883631985 (190h3m8.842729235s) [upload=197828]: Got next segment. Inline: false Aug 16 15:28:46.883695796 (190h3m8.842793047s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:46.883703737 (190h3m8.842800987s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:28:47.148254023 (190h3m9.10735128s) [seg_pos={0 0}, upload=197828]: Begin upload segment... Aug 16 15:28:47.148325960 (190h3m9.107423206s) [seg_pos={0 0}, upload=197828]: Done begin upload segment. Aug 16 15:28:47.148366495 (190h3m9.107463745s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149105833 (190h3m9.108203083s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149659987 (190h3m9.108757238s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150048977 (190h3m9.109146233s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150192097 (190h3m9.109289346s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150487633 (190h3m9.109584885s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150890236 (190h3m9.109987491s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151281889 (190h3m9.11037914s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151630546 (190h3m9.110727799s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151973595 (190h3m9.11107085s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152305026 (190h3m9.111402283s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152634492 (190h3m9.111731742s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.153725357 (190h3m9.112822608s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.154890697 (190h3m9.113987948s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155414981 (190h3m9.114512226s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155964934 (190h3m9.115062197s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.789275832 (190h3m9.748373079s) [upload=197828]: Next returned nil segment Aug 16 15:28:47.789293260 (190h3m9.748390507s) [upload=197828]: Waiting for error group managing segments... Aug 16 15:28:47.942915173 (190h3m9.902012419s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:47.948000865 (190h3m9.907098103s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:52.322732270 (190h3m14.28182951s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:54.226556301 (190h3m16.185653539s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.082436730 (190h3m22.041533992s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.359706605 (190h3m22.318803836s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.641440409 (190h3m22.600537655s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.795956621 (190h3m22.75505386s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:02.308650030 (190h3m24.267747277s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.278922536 (190h3m31.238019793s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.328348624 (190h3m31.287445869s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:10.659976400 (190h3m32.619073639s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983246801 (190h3m35.942344047s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983274710 (190h3m35.942371956s) [seg_pos={0 0}, upload=197828]: Segment reached optimal threshold of 13 pieces. Aug 16 15:29:13.983464694 (190h3m35.942561926s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983650839 (190h3m35.942748071s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection Aug 16 15:29:13.983715300 (190h3m35.942812531s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983728315 (190h3m35.942825563s) [seg_pos={0 0}, upload=197828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:29:13.983839210 (190h3m35.942936459s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:29:13.983862244 (190h3m35.94295949s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:29:14.175384438 (190h3m36.134481686s) [upload=197828]: Done uploading segments... Aug 16 15:29:14.175404292 (190h3m36.134501536s) [upload=197828]: Upload finished. err: <nil> Duration: 14.078352799s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................XX...X..........................X..........] duration: 19.788454207s FAIL (2025-08-16 15:41:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 19.788481912s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...........X..................] duration: 1.868660789s FAIL (2025-08-16 15:41:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.868701897s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 20.98597631s FAIL (2025-08-16 15:40:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 20.98601743s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................XX...X..........................X.........] duration: 14.715946954s FAIL (2025-08-16 15:40:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 14.715975221s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 19.214200249s FAIL (2025-08-16 15:40:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:28:46.883135980 (190h3m8.842233233s) [upload=197828]: Starting upload Aug 16 15:28:46.883238479 (190h3m8.842335732s) [upload=197828]: Done starting upload Aug 16 15:28:46.883377123 (190h3m8.842474378s) [upload=197828]: Uploading segments... Aug 16 15:28:46.883631985 (190h3m8.842729235s) [upload=197828]: Got next segment. Inline: false Aug 16 15:28:46.883695796 (190h3m8.842793047s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:46.883703737 (190h3m8.842800987s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:28:47.148254023 (190h3m9.10735128s) [seg_pos={0 0}, upload=197828]: Begin upload segment... Aug 16 15:28:47.148325960 (190h3m9.107423206s) [seg_pos={0 0}, upload=197828]: Done begin upload segment. Aug 16 15:28:47.148366495 (190h3m9.107463745s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149105833 (190h3m9.108203083s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149659987 (190h3m9.108757238s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150048977 (190h3m9.109146233s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150192097 (190h3m9.109289346s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150487633 (190h3m9.109584885s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150890236 (190h3m9.109987491s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151281889 (190h3m9.11037914s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151630546 (190h3m9.110727799s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151973595 (190h3m9.11107085s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152305026 (190h3m9.111402283s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152634492 (190h3m9.111731742s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.153725357 (190h3m9.112822608s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.154890697 (190h3m9.113987948s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155414981 (190h3m9.114512226s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155964934 (190h3m9.115062197s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.789275832 (190h3m9.748373079s) [upload=197828]: Next returned nil segment Aug 16 15:28:47.789293260 (190h3m9.748390507s) [upload=197828]: Waiting for error group managing segments... Aug 16 15:28:47.942915173 (190h3m9.902012419s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:47.948000865 (190h3m9.907098103s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:52.322732270 (190h3m14.28182951s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:54.226556301 (190h3m16.185653539s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.082436730 (190h3m22.041533992s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.359706605 (190h3m22.318803836s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.641440409 (190h3m22.600537655s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.795956621 (190h3m22.75505386s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:02.308650030 (190h3m24.267747277s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.278922536 (190h3m31.238019793s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.328348624 (190h3m31.287445869s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:10.659976400 (190h3m32.619073639s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983246801 (190h3m35.942344047s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983274710 (190h3m35.942371956s) [seg_pos={0 0}, upload=197828]: Segment reached optimal threshold of 13 pieces. Aug 16 15:29:13.983464694 (190h3m35.942561926s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983650839 (190h3m35.942748071s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection Aug 16 15:29:13.983715300 (190h3m35.942812531s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983728315 (190h3m35.942825563s) [seg_pos={0 0}, upload=197828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:29:13.983839210 (190h3m35.942936459s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:29:13.983862244 (190h3m35.94295949s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:29:14.175384438 (190h3m36.134481686s) [upload=197828]: Done uploading segments... Aug 16 15:29:14.175404292 (190h3m36.134501536s) [upload=197828]: Upload finished. err: <nil> Duration: 19.214248057s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................XX...X..........................X........] duration: 22.407970621s FAIL (2025-08-16 15:40:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 22.408000685s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...........X.................] duration: 1.13038124s FAIL (2025-08-16 15:40:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.130410361s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 28.437857227s FAIL (2025-08-16 15:40:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 28.437891933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................XX...X..........................X.......] duration: 21.75020884s FAIL (2025-08-16 15:40:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 21.75026014s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...........X................] duration: 1.532756842s FAIL (2025-08-16 15:39:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.532816399s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 38.133952523s FAIL (2025-08-16 15:39:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 38.133983864s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 15.113039282s FAIL (2025-08-16 15:39:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:28:46.883135980 (190h3m8.842233233s) [upload=197828]: Starting upload Aug 16 15:28:46.883238479 (190h3m8.842335732s) [upload=197828]: Done starting upload Aug 16 15:28:46.883377123 (190h3m8.842474378s) [upload=197828]: Uploading segments... Aug 16 15:28:46.883631985 (190h3m8.842729235s) [upload=197828]: Got next segment. Inline: false Aug 16 15:28:46.883695796 (190h3m8.842793047s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:46.883703737 (190h3m8.842800987s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:28:47.148254023 (190h3m9.10735128s) [seg_pos={0 0}, upload=197828]: Begin upload segment... Aug 16 15:28:47.148325960 (190h3m9.107423206s) [seg_pos={0 0}, upload=197828]: Done begin upload segment. Aug 16 15:28:47.148366495 (190h3m9.107463745s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149105833 (190h3m9.108203083s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149659987 (190h3m9.108757238s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150048977 (190h3m9.109146233s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150192097 (190h3m9.109289346s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150487633 (190h3m9.109584885s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150890236 (190h3m9.109987491s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151281889 (190h3m9.11037914s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151630546 (190h3m9.110727799s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151973595 (190h3m9.11107085s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152305026 (190h3m9.111402283s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152634492 (190h3m9.111731742s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.153725357 (190h3m9.112822608s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.154890697 (190h3m9.113987948s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155414981 (190h3m9.114512226s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155964934 (190h3m9.115062197s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.789275832 (190h3m9.748373079s) [upload=197828]: Next returned nil segment Aug 16 15:28:47.789293260 (190h3m9.748390507s) [upload=197828]: Waiting for error group managing segments... Aug 16 15:28:47.942915173 (190h3m9.902012419s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:47.948000865 (190h3m9.907098103s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:52.322732270 (190h3m14.28182951s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:54.226556301 (190h3m16.185653539s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.082436730 (190h3m22.041533992s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.359706605 (190h3m22.318803836s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.641440409 (190h3m22.600537655s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.795956621 (190h3m22.75505386s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:02.308650030 (190h3m24.267747277s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.278922536 (190h3m31.238019793s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.328348624 (190h3m31.287445869s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:10.659976400 (190h3m32.619073639s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983246801 (190h3m35.942344047s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983274710 (190h3m35.942371956s) [seg_pos={0 0}, upload=197828]: Segment reached optimal threshold of 13 pieces. Aug 16 15:29:13.983464694 (190h3m35.942561926s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983650839 (190h3m35.942748071s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection Aug 16 15:29:13.983715300 (190h3m35.942812531s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983728315 (190h3m35.942825563s) [seg_pos={0 0}, upload=197828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:29:13.983839210 (190h3m35.942936459s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:29:13.983862244 (190h3m35.94295949s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:29:14.175384438 (190h3m36.134481686s) [upload=197828]: Done uploading segments... Aug 16 15:29:14.175404292 (190h3m36.134501536s) [upload=197828]: Upload finished. err: <nil> Duration: 15.113074722s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................XX...X..........................X......] duration: 12.027401046s FAIL (2025-08-16 15:39:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 12.02743191s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................................X...........X...............] duration: 1.645852685s FAIL (2025-08-16 15:38:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.64589328s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.935704004s FAIL (2025-08-16 15:38:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:28:46.883135980 (190h3m8.842233233s) [upload=197828]: Starting upload Aug 16 15:28:46.883238479 (190h3m8.842335732s) [upload=197828]: Done starting upload Aug 16 15:28:46.883377123 (190h3m8.842474378s) [upload=197828]: Uploading segments... Aug 16 15:28:46.883631985 (190h3m8.842729235s) [upload=197828]: Got next segment. Inline: false Aug 16 15:28:46.883695796 (190h3m8.842793047s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:46.883703737 (190h3m8.842800987s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:28:47.148254023 (190h3m9.10735128s) [seg_pos={0 0}, upload=197828]: Begin upload segment... Aug 16 15:28:47.148325960 (190h3m9.107423206s) [seg_pos={0 0}, upload=197828]: Done begin upload segment. Aug 16 15:28:47.148366495 (190h3m9.107463745s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149105833 (190h3m9.108203083s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149659987 (190h3m9.108757238s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150048977 (190h3m9.109146233s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150192097 (190h3m9.109289346s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150487633 (190h3m9.109584885s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150890236 (190h3m9.109987491s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151281889 (190h3m9.11037914s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151630546 (190h3m9.110727799s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151973595 (190h3m9.11107085s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152305026 (190h3m9.111402283s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152634492 (190h3m9.111731742s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.153725357 (190h3m9.112822608s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.154890697 (190h3m9.113987948s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155414981 (190h3m9.114512226s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155964934 (190h3m9.115062197s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.789275832 (190h3m9.748373079s) [upload=197828]: Next returned nil segment Aug 16 15:28:47.789293260 (190h3m9.748390507s) [upload=197828]: Waiting for error group managing segments... Aug 16 15:28:47.942915173 (190h3m9.902012419s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:47.948000865 (190h3m9.907098103s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:52.322732270 (190h3m14.28182951s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:54.226556301 (190h3m16.185653539s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.082436730 (190h3m22.041533992s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.359706605 (190h3m22.318803836s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.641440409 (190h3m22.600537655s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.795956621 (190h3m22.75505386s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:02.308650030 (190h3m24.267747277s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.278922536 (190h3m31.238019793s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.328348624 (190h3m31.287445869s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:10.659976400 (190h3m32.619073639s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983246801 (190h3m35.942344047s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983274710 (190h3m35.942371956s) [seg_pos={0 0}, upload=197828]: Segment reached optimal threshold of 13 pieces. Aug 16 15:29:13.983464694 (190h3m35.942561926s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983650839 (190h3m35.942748071s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection Aug 16 15:29:13.983715300 (190h3m35.942812531s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983728315 (190h3m35.942825563s) [seg_pos={0 0}, upload=197828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:29:13.983839210 (190h3m35.942936459s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:29:13.983862244 (190h3m35.94295949s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:29:14.175384438 (190h3m36.134481686s) [upload=197828]: Done uploading segments... Aug 16 15:29:14.175404292 (190h3m36.134501536s) [upload=197828]: Upload finished. err: <nil> Duration: 2.935740109s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................XX...X..........................X.....] duration: 32.40773352s FAIL (2025-08-16 15:38:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 32.40775416s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...........X..............] duration: 4.511469104s FAIL (2025-08-16 15:38:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 4.511505453s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 20.901735039s FAIL (2025-08-16 15:38:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 20.901779482s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 50.656789516s FAIL (2025-08-16 15:37:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 50.656818096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................XX...X..........................X....] duration: 50.43325347s FAIL (2025-08-16 15:37:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 50.433290378s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...........X.............] duration: 1.182948251s FAIL (2025-08-16 15:37:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.18298074s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 12.956372531s FAIL (2025-08-16 15:37:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:28:46.883135980 (190h3m8.842233233s) [upload=197828]: Starting upload Aug 16 15:28:46.883238479 (190h3m8.842335732s) [upload=197828]: Done starting upload Aug 16 15:28:46.883377123 (190h3m8.842474378s) [upload=197828]: Uploading segments... Aug 16 15:28:46.883631985 (190h3m8.842729235s) [upload=197828]: Got next segment. Inline: false Aug 16 15:28:46.883695796 (190h3m8.842793047s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:46.883703737 (190h3m8.842800987s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:28:47.148254023 (190h3m9.10735128s) [seg_pos={0 0}, upload=197828]: Begin upload segment... Aug 16 15:28:47.148325960 (190h3m9.107423206s) [seg_pos={0 0}, upload=197828]: Done begin upload segment. Aug 16 15:28:47.148366495 (190h3m9.107463745s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149105833 (190h3m9.108203083s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149659987 (190h3m9.108757238s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150048977 (190h3m9.109146233s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150192097 (190h3m9.109289346s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150487633 (190h3m9.109584885s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150890236 (190h3m9.109987491s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151281889 (190h3m9.11037914s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151630546 (190h3m9.110727799s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151973595 (190h3m9.11107085s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152305026 (190h3m9.111402283s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152634492 (190h3m9.111731742s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.153725357 (190h3m9.112822608s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.154890697 (190h3m9.113987948s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155414981 (190h3m9.114512226s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155964934 (190h3m9.115062197s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.789275832 (190h3m9.748373079s) [upload=197828]: Next returned nil segment Aug 16 15:28:47.789293260 (190h3m9.748390507s) [upload=197828]: Waiting for error group managing segments... Aug 16 15:28:47.942915173 (190h3m9.902012419s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:47.948000865 (190h3m9.907098103s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:52.322732270 (190h3m14.28182951s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:54.226556301 (190h3m16.185653539s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.082436730 (190h3m22.041533992s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.359706605 (190h3m22.318803836s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.641440409 (190h3m22.600537655s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.795956621 (190h3m22.75505386s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:02.308650030 (190h3m24.267747277s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.278922536 (190h3m31.238019793s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.328348624 (190h3m31.287445869s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:10.659976400 (190h3m32.619073639s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983246801 (190h3m35.942344047s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983274710 (190h3m35.942371956s) [seg_pos={0 0}, upload=197828]: Segment reached optimal threshold of 13 pieces. Aug 16 15:29:13.983464694 (190h3m35.942561926s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983650839 (190h3m35.942748071s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection Aug 16 15:29:13.983715300 (190h3m35.942812531s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983728315 (190h3m35.942825563s) [seg_pos={0 0}, upload=197828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:29:13.983839210 (190h3m35.942936459s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:29:13.983862244 (190h3m35.94295949s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:29:14.175384438 (190h3m36.134481686s) [upload=197828]: Done uploading segments... Aug 16 15:29:14.175404292 (190h3m36.134501536s) [upload=197828]: Upload finished. err: <nil> Duration: 12.956437836s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...........X............] duration: 1.438012617s FAIL (2025-08-16 15:37:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 1.438183065s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 30.026029613s FAIL (2025-08-16 15:36:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 30.026059859s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...........X...........] duration: 5.123371613s FAIL (2025-08-16 15:36:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 5.123504852s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.......................XX...X..........................X...] duration: 29.39652828s FAIL (2025-08-16 15:36:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 29.396568365s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 12.544325373s FAIL (2025-08-16 15:36:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:28:46.883135980 (190h3m8.842233233s) [upload=197828]: Starting upload Aug 16 15:28:46.883238479 (190h3m8.842335732s) [upload=197828]: Done starting upload Aug 16 15:28:46.883377123 (190h3m8.842474378s) [upload=197828]: Uploading segments... Aug 16 15:28:46.883631985 (190h3m8.842729235s) [upload=197828]: Got next segment. Inline: false Aug 16 15:28:46.883695796 (190h3m8.842793047s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:46.883703737 (190h3m8.842800987s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:28:47.148254023 (190h3m9.10735128s) [seg_pos={0 0}, upload=197828]: Begin upload segment... Aug 16 15:28:47.148325960 (190h3m9.107423206s) [seg_pos={0 0}, upload=197828]: Done begin upload segment. Aug 16 15:28:47.148366495 (190h3m9.107463745s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149105833 (190h3m9.108203083s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149659987 (190h3m9.108757238s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150048977 (190h3m9.109146233s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150192097 (190h3m9.109289346s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150487633 (190h3m9.109584885s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150890236 (190h3m9.109987491s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151281889 (190h3m9.11037914s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151630546 (190h3m9.110727799s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151973595 (190h3m9.11107085s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152305026 (190h3m9.111402283s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152634492 (190h3m9.111731742s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.153725357 (190h3m9.112822608s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.154890697 (190h3m9.113987948s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155414981 (190h3m9.114512226s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155964934 (190h3m9.115062197s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.789275832 (190h3m9.748373079s) [upload=197828]: Next returned nil segment Aug 16 15:28:47.789293260 (190h3m9.748390507s) [upload=197828]: Waiting for error group managing segments... Aug 16 15:28:47.942915173 (190h3m9.902012419s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:47.948000865 (190h3m9.907098103s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:52.322732270 (190h3m14.28182951s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:54.226556301 (190h3m16.185653539s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.082436730 (190h3m22.041533992s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.359706605 (190h3m22.318803836s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.641440409 (190h3m22.600537655s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.795956621 (190h3m22.75505386s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:02.308650030 (190h3m24.267747277s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.278922536 (190h3m31.238019793s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.328348624 (190h3m31.287445869s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:10.659976400 (190h3m32.619073639s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983246801 (190h3m35.942344047s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983274710 (190h3m35.942371956s) [seg_pos={0 0}, upload=197828]: Segment reached optimal threshold of 13 pieces. Aug 16 15:29:13.983464694 (190h3m35.942561926s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983650839 (190h3m35.942748071s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection Aug 16 15:29:13.983715300 (190h3m35.942812531s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983728315 (190h3m35.942825563s) [seg_pos={0 0}, upload=197828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:29:13.983839210 (190h3m35.942936459s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:29:13.983862244 (190h3m35.94295949s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:29:14.175384438 (190h3m36.134481686s) [upload=197828]: Done uploading segments... Aug 16 15:29:14.175404292 (190h3m36.134501536s) [upload=197828]: Upload finished. err: <nil> Duration: 12.544357058s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.......................XX...X..........................X..] duration: 56.010395059s FAIL (2025-08-16 15:35:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 56.010510733s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 12.385134099s FAIL (2025-08-16 15:35:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 12.385167221s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...........X..........] duration: 3.176702089s FAIL (2025-08-16 15:35:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 3.176732785s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.108108572s FAIL (2025-08-16 15:35:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== tem: pb.BatchRequestItem_SegmentBegin Aug 16 15:34:28.789763009 (190h8m50.748860259s) [seg_pos={0 0}, upload=197930]: Begin upload segment... Aug 16 15:34:28.789838559 (190h8m50.748935804s) [seg_pos={0 0}, upload=197930]: Done begin upload segment. Aug 16 15:34:28.789873820 (190h8m50.748971074s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790028212 (190h8m50.749125466s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790480036 (190h8m50.749577287s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790488463 (190h8m50.749585713s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790924547 (190h8m50.750021801s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.790952481 (190h8m50.750049733s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791356570 (190h8m50.750453816s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791425868 (190h8m50.75052312s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791735223 (190h8m50.750832474s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.791897544 (190h8m50.750994789s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792082620 (190h8m50.751179869s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792400092 (190h8m50.75149734s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792450645 (190h8m50.751547889s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.792906673 (190h8m50.752003908s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793079281 (190h8m50.752176536s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.793640412 (190h8m50.752737664s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794054757 (190h8m50.753152008s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:28.794430707 (190h8m50.753527954s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Uploading piece... Aug 16 15:34:29.603810040 (190h8m51.562907289s) [upload=197930]: Next returned nil segment Aug 16 15:34:29.603852913 (190h8m51.562950163s) [upload=197930]: Waiting for error group managing segments... Aug 16 15:34:29.820565390 (190h8m51.779662649s) [noise=true , address=45.91.116.5:4307 , piece_id=V56DH2M3ZKJDBSM46PU2F45RM2NJUSFIOS65PZZFN3IYKTJFBNBQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ATcyutSAE1A1uN2QcwYFdY1ZvUCZTKiybzXt66NgnqrFYoWWC, address:45.91.116.5:4307): protocol: storage node overloaded, request limit: 2000 Aug 16 15:34:29.977340589 (190h8m51.936437827s) [noise=true , address=77.90.31.15:4700 , piece_id=BNLY4352F5JUTPV2JHMFW7DNJEUC6DBGS3RGJFD4BJTM5KVMWWJQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:30.968018989 (190h8m52.927116228s) [noise=true , address=77.90.31.21:4911 , piece_id=5JV7VPOLN7EE6ITNIWUXC4Y2TO5RP4H4LHLUFCNXNDWGH3IPIS2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:34.731331196 (190h8m56.690428428s) [noise=true , address=79.127.201.215:4359 , piece_id=33YZPBHHHVNTWOQZQRXQYWBKAFVFI4BF5W7XY2WS5UMZ36C45HEA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.053057953 (190h9m0.0121552s) [noise=true , address=66.23.193.175:4553 , piece_id=JT2IAEMLP4FZWXRBCUTR55LZBMCKJ2D2PTJQ7R2SN4RIR5UTZKUA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:38.328734372 (190h9m0.287831613s) [noise=true , address=104.36.82.44:4882 , piece_id=4H2FTYIJ6DDNS4HSGEHDZLCYTYV5RGPQ5LFPGKMHJQPH4L3VRW2Q, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:40.124123016 (190h9m2.083220253s) [noise=true , address=104.36.82.59:4330 , piece_id=435GU443I6WVD5IJE4I57AKHXBOCQYEJHROAYNGHCVOEO6D2YSAA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:55.078023024 (190h9m17.037120258s) [noise=true , address=104.36.82.35:4597 , piece_id=5VTNIMPHWPO6J2MNBTHCO2ONGWZONPU5WLL3NDHHWUUXRUDDXGCQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.616172990 (190h9m19.575270223s) [noise=true , address=104.36.82.60:4352 , piece_id=FSJ3HQE5TWO6PC5TDZ36PAH4USZG5N65EL7NFXL32BXRFZ7SFHRQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:57.808193972 (190h9m19.767291217s) [noise=true , address=104.36.82.39:4759 , piece_id=XD3H4P7F5432NEAR5KXBDWE5O7IQTJPLN7RADRBTJQFDXTVI42XQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:34:58.436040744 (190h9m20.395137978s) [noise=true , address=104.36.82.38:4718 , piece_id=C62OR4NTOU3JAJYUWLI2JMRRPBKEWRN7AHDYTPKFRFB6CB2XSFQQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:11.716816745 (190h9m33.675913991s) [noise=true , address=104.36.82.34:4560 , piece_id=W43DKURLZO7DPKFPS6WKAS7ZUQNOIEPDPMQ34WQELBOPNLPHIR3A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:13.862214320 (190h9m35.821311558s) [noise=true , address=104.36.82.36:4645 , piece_id=TIVOHGCYBVW2YPDCXFDU44COLX7RL6I4L2XNJARTT6UDC36GH2OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: <nil> Aug 16 15:35:28.319944915 (190h9m50.27904216s) [noise=true , address=104.36.82.3:4769 , piece_id=WPNQDUZ4IG7PJBLPEFUJIQ2AND7XWMKMXGXQ4RKYRP23UT55LPTA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: expected piece hash; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; write tcp 139.59.144.22:60810->104.36.82.3:4769: use of closed network connection; EOF Aug 16 15:35:28.320453059 (190h9m50.279550305s) [noise=true , address=77.90.31.11:4556 , piece_id=F7K6MBS2A5ZB55WTHGEG227ZX2ZPEUHIKOV4IR4JDMVPUYT5JBPQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, address:77.90.31.11:4556): protocol: rpc: dial tcp 77.90.31.11:4556: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320518988 (190h9m50.279616234s) [noise=true , address=104.36.82.23:4469 , piece_id=LY2QI7VH2H6OLSRMVYCBRLJCIWZASVNUKNWAPSRFKBQM5ZFZV5OA, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1RWjSXyXqhgufuBewhCBgvBfkerSFeHxJZj88UzusCgiXvibQv, address:104.36.82.23:4469): protocol: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51620->104.36.82.23:4469: use of closed network connection Aug 16 15:35:28.320573009 (190h9m50.279670255s) [noise=true , address=77.90.31.19:4874 , piece_id=BXP34TOPINM2V654KB7MROLRZSNSX5JZQRL3Z6QWDOOYWXUUCA7A, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: read tcp 139.59.144.22:41584->77.90.31.19:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:35:28.320836467 (190h9m50.279933712s) [noise=true , address=77.90.31.13:4659 , piece_id=XX2XFP3BKFVMHDXSK3H6HMB5BPT5E3KKCZJRU3JW2RIZX65PDORQ, seg_pos={0 0}, upload=197930]: Done uploading piece. err: ecclient: upload failed (node:1ScTVpUJBg1xUCzmg1zKsNQ5AZgRbk65md6hgzst9QQ6Vgj4Fc, address:77.90.31.13:4659): protocol: context deadline exceeded; context deadline exceeded Aug 16 15:35:28.320993858 (190h9m50.280091112s) [seg_pos={0 0}, upload=197930]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Aug 16 15:35:28.425637986 (190h9m50.384735221s) [upload=197930]: Done uploading segments... Aug 16 15:35:28.425670206 (190h9m50.384767443s) [upload=197930]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.108138244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...........X.........] duration: 3.710661431s FAIL (2025-08-16 15:35:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 3.71071757s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 3.150217846s FAIL (2025-08-16 15:35:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== Aug 16 15:28:46.883135980 (190h3m8.842233233s) [upload=197828]: Starting upload Aug 16 15:28:46.883238479 (190h3m8.842335732s) [upload=197828]: Done starting upload Aug 16 15:28:46.883377123 (190h3m8.842474378s) [upload=197828]: Uploading segments... Aug 16 15:28:46.883631985 (190h3m8.842729235s) [upload=197828]: Got next segment. Inline: false Aug 16 15:28:46.883695796 (190h3m8.842793047s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:46.883703737 (190h3m8.842800987s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:28:47.148254023 (190h3m9.10735128s) [seg_pos={0 0}, upload=197828]: Begin upload segment... Aug 16 15:28:47.148325960 (190h3m9.107423206s) [seg_pos={0 0}, upload=197828]: Done begin upload segment. Aug 16 15:28:47.148366495 (190h3m9.107463745s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149105833 (190h3m9.108203083s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.149659987 (190h3m9.108757238s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150048977 (190h3m9.109146233s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150192097 (190h3m9.109289346s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150487633 (190h3m9.109584885s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.150890236 (190h3m9.109987491s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151281889 (190h3m9.11037914s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151630546 (190h3m9.110727799s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.151973595 (190h3m9.11107085s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152305026 (190h3m9.111402283s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.152634492 (190h3m9.111731742s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.153725357 (190h3m9.112822608s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.154890697 (190h3m9.113987948s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155414981 (190h3m9.114512226s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.155964934 (190h3m9.115062197s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Uploading piece... Aug 16 15:28:47.789275832 (190h3m9.748373079s) [upload=197828]: Next returned nil segment Aug 16 15:28:47.789293260 (190h3m9.748390507s) [upload=197828]: Waiting for error group managing segments... Aug 16 15:28:47.942915173 (190h3m9.902012419s) [noise=true , address=104.36.82.7:4909 , piece_id=5M6VMON32TIP7COFASYWNPUP3WKRMJCORTRBVYAA7ERBS34IWKQQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:47.948000865 (190h3m9.907098103s) [noise=true , address=104.36.82.12:4149 , piece_id=PMSGJNOK54RMHEWZGIDRKVSYOVFMDWITL7WMYFZKAHORQTIL54MQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:52.322732270 (190h3m14.28182951s) [noise=true , address=104.36.82.8:4969 , piece_id=PGI2FCEB7K47JBHNAQZ2SKQIE4JNAXS37F3SP542YLI723SQQQMA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:28:54.226556301 (190h3m16.185653539s) [noise=true , address=77.90.31.10:4517 , piece_id=M3JH24CTU6XYAAID6U6NXEEOYRE3UTPJNUOY5G2YKWSVWMMXDFJQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.082436730 (190h3m22.041533992s) [noise=true , address=45.91.116.12:4227 , piece_id=5KRWPQKIXVFQI4KNP4EF3ZN77SK76QHRQPOMWGDD6GBHFBNCJANA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.359706605 (190h3m22.318803836s) [noise=true , address=45.91.116.8:4048 , piece_id=MQHV75M5DSEZVKG33MPO726W6CJR5B45RZGHOVSW26UCUZTM55XA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.641440409 (190h3m22.600537655s) [noise=true , address=104.36.82.11:4113 , piece_id=47Y4ENCD46CHSGC6XAF24V7IUCNHRHOQ4ZK2XUSVA7PIY4TPP4KA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:00.795956621 (190h3m22.75505386s) [noise=true , address=45.91.116.7:4011 , piece_id=T42O522DV46LPSG6OX4LTDTKRIYBMJNA4OZPD4ZA3DP57NIAKFXQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:02.308650030 (190h3m24.267747277s) [noise=true , address=104.36.82.11:4115 , piece_id=4BQ4S73CPTYIA4HRJLHE42BAXIWN7DX4RF6DR5LY7WYE73FF4JBA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.278922536 (190h3m31.238019793s) [noise=true , address=45.91.116.4:4938 , piece_id=GDPMV64A2HFFOY3PGHWBJU2U37CBCSTZSVRH4COTFMMK74ZZS4ZA, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:09.328348624 (190h3m31.287445869s) [noise=true , address=77.90.31.11:4554 , piece_id=Q65UBZG44O7VCWGY7O32PEO7HL5G2EIRSLMLRKOYPAFYGZ2GWQFQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:10.659976400 (190h3m32.619073639s) [noise=true , address=45.91.116.11:4119 , piece_id=77KB4LU3U5YOZHLVR7G6L3LLUNZV6ELBZZIU2HDFIXHXYAIOT3QQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983246801 (190h3m35.942344047s) [noise=true , address=104.36.82.3:4767 , piece_id=Y6ZVKSJ7P3H4W7QBKNP7N53UNC64GD6BGE6ADBPVNHNQNGWCOVZQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: <nil> Aug 16 15:29:13.983274710 (190h3m35.942371956s) [seg_pos={0 0}, upload=197828]: Segment reached optimal threshold of 13 pieces. Aug 16 15:29:13.983464694 (190h3m35.942561926s) [noise=true , address=104.36.82.5:4839 , piece_id=U47HEVNWETNTLFMFZOJFLNK4TXZMPKJ4KQWYN2Q2KIGFN43J3RVQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983650839 (190h3m35.942748071s) [noise=true , address=77.90.31.13:4627 , piece_id=VL4PT6FZFYUCKGTHDG5V3IAEECKFI75CSU2AIGGVQGVY6SRFSHYQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:49782->77.90.31.13:4627: use of closed network connection Aug 16 15:29:13.983715300 (190h3m35.942812531s) [noise=true , address=77.90.31.14:4662 , piece_id=HJCRSIH7WW7UHX4T3B6E367BI4UFB5KARICU7ILKZPJ6BXJIPYTQ, seg_pos={0 0}, upload=197828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Aug 16 15:29:13.983728315 (190h3m35.942825563s) [seg_pos={0 0}, upload=197828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 16 15:29:13.983839210 (190h3m35.942936459s) [upload=197828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:29:13.983862244 (190h3m35.94295949s) [upload=197828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:29:14.175384438 (190h3m36.134481686s) [upload=197828]: Done uploading segments... Aug 16 15:29:14.175404292 (190h3m36.134501536s) [upload=197828]: Upload finished. err: <nil> Duration: 3.150255029s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.......................XX...X..........................X.] duration: 32.333763383s FAIL (2025-08-16 15:34:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: invalid range 0:4194304 (size:4194304) ======LOGS====== Aug 16 15:33:12.429086776 (190h7m34.388184027s) [upload=197909]: Starting upload Aug 16 15:33:12.429151496 (190h7m34.388248739s) [upload=197909]: Done starting upload Aug 16 15:33:12.429444972 (190h7m34.388542209s) [upload=197909]: Uploading segments... Aug 16 15:33:12.429713475 (190h7m34.388810714s) [upload=197909]: Got next segment. Inline: false Aug 16 15:33:12.429768469 (190h7m34.388865704s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:33:12.429773813 (190h7m34.388871061s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 16 15:33:13.196073678 (190h7m35.155170931s) [seg_pos={0 0}, upload=197909]: Begin upload segment... Aug 16 15:33:13.196123786 (190h7m35.155221017s) [seg_pos={0 0}, upload=197909]: Done begin upload segment. Aug 16 15:33:13.196156637 (190h7m35.155253888s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196167842 (190h7m35.155265094s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196509667 (190h7m35.155606915s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196652848 (190h7m35.155750096s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.196746150 (190h7m35.155843398s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197123495 (190h7m35.156220752s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197181198 (190h7m35.156278446s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197456697 (190h7m35.156553949s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197545912 (190h7m35.156643224s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.197740966 (190h7m35.156838212s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198090847 (190h7m35.157188099s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198392729 (190h7m35.15748998s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198683136 (190h7m35.157780495s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198858103 (190h7m35.157955348s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.198949582 (190h7m35.15804683s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199182885 (190h7m35.158280133s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199407286 (190h7m35.15850453s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.199588434 (190h7m35.158685673s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Uploading piece... Aug 16 15:33:13.298700967 (190h7m35.257798216s) [noise=true , address=77.90.31.19:4855 , piece_id=H67KWKLAKLQSX43UNLMLV645UIUFZ2A6JDQRHRATUXIRFZRKYMYQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: ecclient: upload failed (node:19LE2Lq4e6HojnMp9xQ3ySLkRe5PemXTwiQZThN5jT46bQhN6K, address:77.90.31.19:4855): protocol: write tcp 139.59.144.22:56824->77.90.31.19:4855: write: connection refused; conn closed: op; piecestore: piecestore close: conn closed: op Aug 16 15:33:13.963714407 (190h7m35.922811652s) [upload=197909]: Next returned nil segment Aug 16 15:33:13.963733497 (190h7m35.922830744s) [upload=197909]: Waiting for error group managing segments... Aug 16 15:33:14.581779474 (190h7m36.540876729s) [noise=true , address=77.90.31.15:4708 , piece_id=G5HZIV3IYBIE35YQJ3DI5DPYJLHYBQBDCJDANAM3SAJDQHYYPDQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:14.710418847 (190h7m36.669516083s) [noise=true , address=77.90.31.13:4652 , piece_id=OZBK2MWDTLWHBA6UZJVPRLFL4KDMIVC7YYQEKUIAVQIT54M3LR7Q, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.374088154 (190h7m37.333185399s) [noise=true , address=79.127.220.101:4075 , piece_id=6I4QM5TEXJ7JOSJTAJAYRH6MMRATKSFCQTRG2OMQURBPWKQMPDFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:15.471621996 (190h7m37.430719243s) [noise=true , address=104.36.82.5:4856 , piece_id=XK4QM2XWSD6XY2T7NEEIRVGDHBRDIZ7BCJZOJ7VLEIOHZRT2QMXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:16.310885235 (190h7m38.26998248s) [noise=true , address=79.127.201.215:4351 , piece_id=QFUWK5JTFBPE53DHGJHCUHS774BCIBAOM3HXK7Z3KT2GGK4NINWA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.320374766 (190h7m42.279472012s) [noise=true , address=104.36.82.35:4581 , piece_id=GGE3PRQK5UWLESXBKY25FSOBS3VOUKA4455A5BG3DSCUWQD2MEXA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:20.556789979 (190h7m42.515887231s) [noise=true , address=104.36.82.16:4322 , piece_id=QCTUCR36BMCDNGUR5N72YWXEUE5II5GOXPVWX64RMVGGVDZ35PQA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.241652208 (190h7m44.200749439s) [noise=true , address=104.36.82.4:4830 , piece_id=EMAFFUPD6W2G7FSIT4NYQBXVDCMGPXPCVOZVFMIU3DLN2SUJAYVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:22.685463947 (190h7m44.644561183s) [noise=true , address=104.36.82.38:4697 , piece_id=SWQJ6XUETFATA2SDAMVXWGPT5WDES7XWV4SBSKLWPQTG2HX27MVQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:23.146284913 (190h7m45.105382159s) [noise=true , address=104.36.82.36:4634 , piece_id=66UQQJHPLZSSQP26HJJZS3YPYNQNSFSSG7AG5LP7XTY6STY77OBA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:25.363617620 (190h7m47.322714866s) [noise=true , address=104.36.82.34:4579 , piece_id=K4O4PHXEMLMWF6DFZWQTYIMWDVHB2UGLF5LYTVLZIOUPF2RTJVQQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.187144555 (190h7m48.14624179s) [noise=true , address=66.23.193.175:4557 , piece_id=QBEUJVOO7SFY3GCYTRP5Y4UP3YVUFKQYUCCWSXU2SJHCXZUG3GDA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:26.949174295 (190h7m48.908271528s) [noise=true , address=104.36.82.60:4351 , piece_id=DADS23UIXPGIXZIEEC2A4OHSYLWVJDAIGXT6L4C2QCLWXJDPFCFA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818239471 (190h7m51.777336722s) [noise=true , address=104.36.82.2:4756 , piece_id=UOP64KVPHQGJ2VIVBCB7MYP3E6QIA5CTQ7HLCLRXQEO4U3ZTZDNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: <nil> Aug 16 15:33:29.818280074 (190h7m51.777377322s) [seg_pos={0 0}, upload=197909]: Segment reached optimal threshold of 14 pieces. Aug 16 15:33:29.818445418 (190h7m51.777542654s) [noise=true , address=104.36.82.39:4730 , piece_id=BASBQF7GQ3UFBYK4TOISAHYEABHEEY6CDAX6P42DXATZN4XZBUNQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818488194 (190h7m51.777585426s) [noise=true , address=104.36.82.44:4870 , piece_id=NQKSG72ZAP3GCSJNR4VLENQCW3P4ZMT2AN2J6G35UIV5RH2HR2EA, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Aug 16 15:33:29.818612882 (190h7m51.777710118s) [noise=true , address=77.90.31.21:4898 , piece_id=DCFVQBX7FIBKV7A6MKBZQG44UBLGIXEO5KIQN4ETQAMTS4VDV6WQ, seg_pos={0 0}, upload=197909]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qi796rSqngMJNBiFji2g4eYRn5MsqCqhpGPdYPN2GN8yFGTUx): protocol: write tcp 139.59.144.22:39702->77.90.31.21:4898: use of closed network connection; conn closed: op Aug 16 15:33:29.818643481 (190h7m51.777740729s) [seg_pos={0 0}, upload=197909]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 16 15:33:29.818733041 (190h7m51.777830276s) [upload=197909]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 16 15:33:29.818738378 (190h7m51.777835625s) [upload=197909]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:33:32.488459624 (190h7m54.447556871s) [upload=197909]: Done uploading segments... Aug 16 15:33:32.488477260 (190h7m54.447574499s) [upload=197909]: Upload finished. err: <nil> Duration: 32.333789096s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...........X........] duration: 821.254368ms FAIL (2025-08-16 15:34:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 821.288967ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...........X.......] duration: 7.759299683s FAIL (2025-08-16 15:34:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/1K):: after f3fccd73a10c40d4c496ae7b765b67b3e313c03ad8908ca9177441689430c299 != before 67df91a5564985574444a27c5c6cba1d7efc11f5dc2efbc83de02c011a1e86ec ======LOGS====== Aug 16 15:28:52.206520610 (190h3m14.16561787s) [upload=197832]: Starting upload Aug 16 15:28:52.206624583 (190h3m14.165721839s) [upload=197832]: Done starting upload Aug 16 15:28:52.206719666 (190h3m14.165816898s) [upload=197832]: Uploading segments... Aug 16 15:28:52.206788764 (190h3m14.16588602s) [upload=197832]: Got next segment. Inline: true Aug 16 15:28:52.206795487 (190h3m14.165892742s) [upload=197832]: Waiting for error group managing segments... Aug 16 15:28:52.206864905 (190h3m14.165962162s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 16 15:28:52.206870689 (190h3m14.16596794s) [upload=197832]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 16 15:28:52.206875010 (190h3m14.165972261s) [upload=197832]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 16 15:28:54.637176780 (190h3m16.596274033s) [upload=197832]: Done uploading segments... Aug 16 15:28:54.637197785 (190h3m16.596295022s) [upload=197832]: Upload finished. err: <nil> Duration:* 7.759346077s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)