Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
13.435352189s trace:
https://jaeger.dev.storj.io/trace/6e0ed516680ebec8
FAIL (2026-01-06 20:08:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A) context deadline exceeded
======LOGS======
Jan 6 19:58:17.403555994 (772h29m57.383908645s) [upload=679917]: Starting upload
Jan 6 19:58:17.403666206 (772h29m57.384018858s) [upload=679917]: Done starting upload
Jan 6 19:58:17.403842502 (772h29m57.38419514s) [upload=679917]: Uploading segments...
Jan 6 19:58:17.404166965 (772h29m57.384519606s) [upload=679917]: Got next segment. Inline: false
Jan 6 19:58:17.404223968 (772h29m57.38457663s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 19:58:17.404230594 (772h29m57.384583235s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 19:58:17.516709204 (772h29m57.49706185s) [seg_pos={0 0}, upload=679917]: Begin upload segment...
Jan 6 19:58:17.516795101 (772h29m57.497147747s) [seg_pos={0 0}, upload=679917]: Done begin upload segment.
Jan 6 19:58:17.516832647 (772h29m57.497185292s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.517030264 (772h29m57.497382912s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518092013 (772h29m57.498444671s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518958504 (772h29m57.49931115s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519239753 (772h29m57.4995924s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519449404 (772h29m57.499802051s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519955276 (772h29m57.500307924s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520193176 (772h29m57.500545822s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520785527 (772h29m57.501138173s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521208368 (772h29m57.501561018s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521330375 (772h29m57.501683021s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521967075 (772h29m57.502319718s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522287136 (772h29m57.502639784s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522546502 (772h29m57.502899148s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.523557905 (772h29m57.50391057s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.524586930 (772h29m57.504939577s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:18.166215094 (772h29m58.146567733s) [upload=679917]: Next returned nil segment
Jan 6 19:58:18.166228823 (772h29m58.14658146s) [upload=679917]: Waiting for error group managing segments...
Jan 6 19:58:18.302286606 (772h29m58.282639244s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.343646848 (772h29m58.323999484s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.357722828 (772h29m58.338075467s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.367003674 (772h29m58.347356311s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.376262781 (772h29m58.356615421s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.403506818 (772h29m58.383859465s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.465404723 (772h29m58.445757372s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.567776867 (772h29m58.548129504s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.599132864 (772h29m58.579485506s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.613759829 (772h29m58.594112475s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.664243429 (772h29m58.644596056s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.895647542 (772h29m58.876000181s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102892198 (772h29m59.08324489s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102980997 (772h29m59.083333641s) [seg_pos={0 0}, upload=679917]: Segment reached optimal threshold of 13 pieces.
Jan 6 19:58:19.103439612 (772h29m59.083792261s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.103456166 (772h29m59.083808814s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.103732510 (772h29m59.084085155s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection
Jan 6 19:58:19.103739973 (772h29m59.084092617s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104075008 (772h29m59.084427655s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.104083229 (772h29m59.08443587s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104103417 (772h29m59.084456044s) [seg_pos={0 0}, upload=679917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 19:58:19.104195361 (772h29m59.084548002s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:19.104203262 (772h29m59.08455589s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:19.248322143 (772h29m59.228674781s) [upload=679917]: Done uploading segments...
Jan 6 19:58:19.248344948 (772h29m59.228697584s) [upload=679917]: Upload finished. err: <nil>
Duration: 13.435386816s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
3.130936407s trace:
https://jaeger.dev.storj.io/trace/5cf4e6a4e2080b2d
FAIL (2026-01-06 20:07:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
4 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.233875032 (772h29m46.214227679s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234056119 (772h29m46.21440876s) [seg_pos={0 0}, upload=679912]: Done begin upload segment.
Jan 6 19:58:06.234140741 (772h29m46.214493379s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234498043 (772h29m46.214850694s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234792560 (772h29m46.215145197s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235003281 (772h29m46.215355922s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235277131 (772h29m46.215629772s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235848449 (772h29m46.216201078s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.236290164 (772h29m46.216642812s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237033388 (772h29m46.217386026s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237491382 (772h29m46.217844022s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.238084906 (772h29m46.218437545s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.239186345 (772h29m46.219538984s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.240506522 (772h29m46.220859165s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.829743229 (772h29m46.810095872s) [upload=679912]: Next returned nil segment
Jan 6 19:58:06.829762199 (772h29m46.810114841s) [upload=679912]: Waiting for error group managing segments...
Jan 6 19:58:06.947371887 (772h29m46.927724525s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.002507087 (772h29m46.982859763s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.039550016 (772h29m47.019902653s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.073221490 (772h29m47.053574126s) [noise=true , address=77.90.31.18:4834 , piece_id=IXY6FIN7YKAX7HIE6YYF6C3CT6JQB7OCD5YQA2R3NXPV7ZSVNO5A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.121426552 (772h29m47.101779189s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.147306902 (772h29m47.127659539s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.155135219 (772h29m47.135487858s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.307741749 (772h29m47.288094384s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.440337127 (772h29m47.420689763s) [noise=true , address=45.91.116.13:4184 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.901506940 (772h29m47.88185958s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.932347050 (772h29m47.912699689s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.046430498 (772h29m48.026783134s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.063258213 (772h29m48.043610858s) [noise=true , address=145.79.227.14:4866 , piece_id=NMEQDY2COSGV6K4JIYXEYQ2A66BJRRCOAGCFS2URAOWXUB23QM4Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218417658 (772h29m48.198770297s) [noise=true , address=145.79.227.30:4789 , piece_id=UYSB4JQOVVVK6LYXE7A72MBUU4VGT7TAXXVGNTWY25JMO4OGK3WQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218446170 (772h29m48.198798807s) [seg_pos={0 0}, upload=679912]: Segment reached optimal threshold of 14 pieces.
Jan 6 19:58:08.218679983 (772h29m48.199032625s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection
Jan 6 19:58:08.218688530 (772h29m48.199041166s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218829688 (772h29m48.199182333s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:08.218854097 (772h29m48.19920674s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218942630 (772h29m48.199295275s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: read tcp 139.59.144.22:56426->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 19:58:08.218950936 (772h29m48.199303572s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219048646 (772h29m48.199401291s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection
Jan 6 19:58:08.219057565 (772h29m48.199410201s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219089322 (772h29m48.199441949s) [seg_pos={0 0}, upload=679912]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 19:58:08.219237200 (772h29m48.199589839s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:08.219244038 (772h29m48.199596674s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:08.352159986 (772h29m48.332512623s) [upload=679912]: Done uploading segments...
Jan 6 19:58:08.352210140 (772h29m48.332562776s) [upload=679912]: Upload finished. err: <nil>
Duration: 3.130993576s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
2.369054076s trace:
https://jaeger.dev.storj.io/trace/6e0ed516680ebec8
FAIL (2026-01-06 20:07:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A) context deadline exceeded
======LOGS======
Jan 6 19:58:17.403555994 (772h29m57.383908645s) [upload=679917]: Starting upload
Jan 6 19:58:17.403666206 (772h29m57.384018858s) [upload=679917]: Done starting upload
Jan 6 19:58:17.403842502 (772h29m57.38419514s) [upload=679917]: Uploading segments...
Jan 6 19:58:17.404166965 (772h29m57.384519606s) [upload=679917]: Got next segment. Inline: false
Jan 6 19:58:17.404223968 (772h29m57.38457663s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 19:58:17.404230594 (772h29m57.384583235s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 19:58:17.516709204 (772h29m57.49706185s) [seg_pos={0 0}, upload=679917]: Begin upload segment...
Jan 6 19:58:17.516795101 (772h29m57.497147747s) [seg_pos={0 0}, upload=679917]: Done begin upload segment.
Jan 6 19:58:17.516832647 (772h29m57.497185292s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.517030264 (772h29m57.497382912s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518092013 (772h29m57.498444671s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518958504 (772h29m57.49931115s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519239753 (772h29m57.4995924s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519449404 (772h29m57.499802051s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519955276 (772h29m57.500307924s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520193176 (772h29m57.500545822s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520785527 (772h29m57.501138173s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521208368 (772h29m57.501561018s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521330375 (772h29m57.501683021s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521967075 (772h29m57.502319718s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522287136 (772h29m57.502639784s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522546502 (772h29m57.502899148s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.523557905 (772h29m57.50391057s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.524586930 (772h29m57.504939577s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:18.166215094 (772h29m58.146567733s) [upload=679917]: Next returned nil segment
Jan 6 19:58:18.166228823 (772h29m58.14658146s) [upload=679917]: Waiting for error group managing segments...
Jan 6 19:58:18.302286606 (772h29m58.282639244s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.343646848 (772h29m58.323999484s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.357722828 (772h29m58.338075467s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.367003674 (772h29m58.347356311s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.376262781 (772h29m58.356615421s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.403506818 (772h29m58.383859465s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.465404723 (772h29m58.445757372s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.567776867 (772h29m58.548129504s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.599132864 (772h29m58.579485506s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.613759829 (772h29m58.594112475s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.664243429 (772h29m58.644596056s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.895647542 (772h29m58.876000181s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102892198 (772h29m59.08324489s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102980997 (772h29m59.083333641s) [seg_pos={0 0}, upload=679917]: Segment reached optimal threshold of 13 pieces.
Jan 6 19:58:19.103439612 (772h29m59.083792261s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.103456166 (772h29m59.083808814s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.103732510 (772h29m59.084085155s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection
Jan 6 19:58:19.103739973 (772h29m59.084092617s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104075008 (772h29m59.084427655s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.104083229 (772h29m59.08443587s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104103417 (772h29m59.084456044s) [seg_pos={0 0}, upload=679917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 19:58:19.104195361 (772h29m59.084548002s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:19.104203262 (772h29m59.08455589s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:19.248322143 (772h29m59.228674781s) [upload=679917]: Done uploading segments...
Jan 6 19:58:19.248344948 (772h29m59.228697584s) [upload=679917]: Upload finished. err: <nil>
Duration: 2.369079743s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
12.523893262s trace:
https://jaeger.dev.storj.io/trace/5cf4e6a4e2080b2d
FAIL (2026-01-06 20:06:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
4 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.233875032 (772h29m46.214227679s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234056119 (772h29m46.21440876s) [seg_pos={0 0}, upload=679912]: Done begin upload segment.
Jan 6 19:58:06.234140741 (772h29m46.214493379s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234498043 (772h29m46.214850694s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234792560 (772h29m46.215145197s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235003281 (772h29m46.215355922s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235277131 (772h29m46.215629772s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235848449 (772h29m46.216201078s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.236290164 (772h29m46.216642812s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237033388 (772h29m46.217386026s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237491382 (772h29m46.217844022s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.238084906 (772h29m46.218437545s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.239186345 (772h29m46.219538984s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.240506522 (772h29m46.220859165s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.829743229 (772h29m46.810095872s) [upload=679912]: Next returned nil segment
Jan 6 19:58:06.829762199 (772h29m46.810114841s) [upload=679912]: Waiting for error group managing segments...
Jan 6 19:58:06.947371887 (772h29m46.927724525s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.002507087 (772h29m46.982859763s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.039550016 (772h29m47.019902653s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.073221490 (772h29m47.053574126s) [noise=true , address=77.90.31.18:4834 , piece_id=IXY6FIN7YKAX7HIE6YYF6C3CT6JQB7OCD5YQA2R3NXPV7ZSVNO5A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.121426552 (772h29m47.101779189s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.147306902 (772h29m47.127659539s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.155135219 (772h29m47.135487858s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.307741749 (772h29m47.288094384s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.440337127 (772h29m47.420689763s) [noise=true , address=45.91.116.13:4184 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.901506940 (772h29m47.88185958s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.932347050 (772h29m47.912699689s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.046430498 (772h29m48.026783134s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.063258213 (772h29m48.043610858s) [noise=true , address=145.79.227.14:4866 , piece_id=NMEQDY2COSGV6K4JIYXEYQ2A66BJRRCOAGCFS2URAOWXUB23QM4Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218417658 (772h29m48.198770297s) [noise=true , address=145.79.227.30:4789 , piece_id=UYSB4JQOVVVK6LYXE7A72MBUU4VGT7TAXXVGNTWY25JMO4OGK3WQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218446170 (772h29m48.198798807s) [seg_pos={0 0}, upload=679912]: Segment reached optimal threshold of 14 pieces.
Jan 6 19:58:08.218679983 (772h29m48.199032625s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection
Jan 6 19:58:08.218688530 (772h29m48.199041166s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218829688 (772h29m48.199182333s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:08.218854097 (772h29m48.19920674s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218942630 (772h29m48.199295275s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: read tcp 139.59.144.22:56426->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 19:58:08.218950936 (772h29m48.199303572s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219048646 (772h29m48.199401291s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection
Jan 6 19:58:08.219057565 (772h29m48.199410201s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219089322 (772h29m48.199441949s) [seg_pos={0 0}, upload=679912]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 19:58:08.219237200 (772h29m48.199589839s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:08.219244038 (772h29m48.199596674s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:08.352159986 (772h29m48.332512623s) [upload=679912]: Done uploading segments...
Jan 6 19:58:08.352210140 (772h29m48.332562776s) [upload=679912]: Upload finished. err: <nil>
Duration: 12.523997237s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
42.461313596s trace:
https://jaeger.dev.storj.io/trace/5cf4e6a4e2080b2d
FAIL (2026-01-06 20:06:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
4 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.233875032 (772h29m46.214227679s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234056119 (772h29m46.21440876s) [seg_pos={0 0}, upload=679912]: Done begin upload segment.
Jan 6 19:58:06.234140741 (772h29m46.214493379s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234498043 (772h29m46.214850694s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234792560 (772h29m46.215145197s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235003281 (772h29m46.215355922s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235277131 (772h29m46.215629772s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235848449 (772h29m46.216201078s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.236290164 (772h29m46.216642812s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237033388 (772h29m46.217386026s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237491382 (772h29m46.217844022s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.238084906 (772h29m46.218437545s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.239186345 (772h29m46.219538984s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.240506522 (772h29m46.220859165s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.829743229 (772h29m46.810095872s) [upload=679912]: Next returned nil segment
Jan 6 19:58:06.829762199 (772h29m46.810114841s) [upload=679912]: Waiting for error group managing segments...
Jan 6 19:58:06.947371887 (772h29m46.927724525s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.002507087 (772h29m46.982859763s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.039550016 (772h29m47.019902653s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.073221490 (772h29m47.053574126s) [noise=true , address=77.90.31.18:4834 , piece_id=IXY6FIN7YKAX7HIE6YYF6C3CT6JQB7OCD5YQA2R3NXPV7ZSVNO5A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.121426552 (772h29m47.101779189s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.147306902 (772h29m47.127659539s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.155135219 (772h29m47.135487858s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.307741749 (772h29m47.288094384s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.440337127 (772h29m47.420689763s) [noise=true , address=45.91.116.13:4184 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.901506940 (772h29m47.88185958s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.932347050 (772h29m47.912699689s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.046430498 (772h29m48.026783134s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.063258213 (772h29m48.043610858s) [noise=true , address=145.79.227.14:4866 , piece_id=NMEQDY2COSGV6K4JIYXEYQ2A66BJRRCOAGCFS2URAOWXUB23QM4Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218417658 (772h29m48.198770297s) [noise=true , address=145.79.227.30:4789 , piece_id=UYSB4JQOVVVK6LYXE7A72MBUU4VGT7TAXXVGNTWY25JMO4OGK3WQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218446170 (772h29m48.198798807s) [seg_pos={0 0}, upload=679912]: Segment reached optimal threshold of 14 pieces.
Jan 6 19:58:08.218679983 (772h29m48.199032625s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection
Jan 6 19:58:08.218688530 (772h29m48.199041166s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218829688 (772h29m48.199182333s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:08.218854097 (772h29m48.19920674s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218942630 (772h29m48.199295275s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: read tcp 139.59.144.22:56426->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 19:58:08.218950936 (772h29m48.199303572s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219048646 (772h29m48.199401291s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection
Jan 6 19:58:08.219057565 (772h29m48.199410201s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219089322 (772h29m48.199441949s) [seg_pos={0 0}, upload=679912]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 19:58:08.219237200 (772h29m48.199589839s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:08.219244038 (772h29m48.199596674s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:08.352159986 (772h29m48.332512623s) [upload=679912]: Done uploading segments...
Jan 6 19:58:08.352210140 (772h29m48.332562776s) [upload=679912]: Upload finished. err: <nil>
Duration: 42.461404916s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
2.492118244s trace:
https://jaeger.dev.storj.io/trace/6e0ed516680ebec8
FAIL (2026-01-06 20:06:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A) context deadline exceeded
======LOGS======
Jan 6 19:58:17.403555994 (772h29m57.383908645s) [upload=679917]: Starting upload
Jan 6 19:58:17.403666206 (772h29m57.384018858s) [upload=679917]: Done starting upload
Jan 6 19:58:17.403842502 (772h29m57.38419514s) [upload=679917]: Uploading segments...
Jan 6 19:58:17.404166965 (772h29m57.384519606s) [upload=679917]: Got next segment. Inline: false
Jan 6 19:58:17.404223968 (772h29m57.38457663s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 19:58:17.404230594 (772h29m57.384583235s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 19:58:17.516709204 (772h29m57.49706185s) [seg_pos={0 0}, upload=679917]: Begin upload segment...
Jan 6 19:58:17.516795101 (772h29m57.497147747s) [seg_pos={0 0}, upload=679917]: Done begin upload segment.
Jan 6 19:58:17.516832647 (772h29m57.497185292s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.517030264 (772h29m57.497382912s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518092013 (772h29m57.498444671s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518958504 (772h29m57.49931115s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519239753 (772h29m57.4995924s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519449404 (772h29m57.499802051s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519955276 (772h29m57.500307924s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520193176 (772h29m57.500545822s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520785527 (772h29m57.501138173s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521208368 (772h29m57.501561018s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521330375 (772h29m57.501683021s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521967075 (772h29m57.502319718s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522287136 (772h29m57.502639784s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522546502 (772h29m57.502899148s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.523557905 (772h29m57.50391057s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.524586930 (772h29m57.504939577s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:18.166215094 (772h29m58.146567733s) [upload=679917]: Next returned nil segment
Jan 6 19:58:18.166228823 (772h29m58.14658146s) [upload=679917]: Waiting for error group managing segments...
Jan 6 19:58:18.302286606 (772h29m58.282639244s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.343646848 (772h29m58.323999484s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.357722828 (772h29m58.338075467s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.367003674 (772h29m58.347356311s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.376262781 (772h29m58.356615421s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.403506818 (772h29m58.383859465s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.465404723 (772h29m58.445757372s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.567776867 (772h29m58.548129504s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.599132864 (772h29m58.579485506s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.613759829 (772h29m58.594112475s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.664243429 (772h29m58.644596056s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.895647542 (772h29m58.876000181s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102892198 (772h29m59.08324489s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102980997 (772h29m59.083333641s) [seg_pos={0 0}, upload=679917]: Segment reached optimal threshold of 13 pieces.
Jan 6 19:58:19.103439612 (772h29m59.083792261s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.103456166 (772h29m59.083808814s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.103732510 (772h29m59.084085155s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection
Jan 6 19:58:19.103739973 (772h29m59.084092617s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104075008 (772h29m59.084427655s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.104083229 (772h29m59.08443587s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104103417 (772h29m59.084456044s) [seg_pos={0 0}, upload=679917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 19:58:19.104195361 (772h29m59.084548002s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:19.104203262 (772h29m59.08455589s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:19.248322143 (772h29m59.228674781s) [upload=679917]: Done uploading segments...
Jan 6 19:58:19.248344948 (772h29m59.228697584s) [upload=679917]: Upload finished. err: <nil>
Duration: 2.49214504s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
3.15577047s trace:
https://jaeger.dev.storj.io/trace/5cf4e6a4e2080b2d
FAIL (2026-01-06 20:05:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
4 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.233875032 (772h29m46.214227679s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234056119 (772h29m46.21440876s) [seg_pos={0 0}, upload=679912]: Done begin upload segment.
Jan 6 19:58:06.234140741 (772h29m46.214493379s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234498043 (772h29m46.214850694s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234792560 (772h29m46.215145197s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235003281 (772h29m46.215355922s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235277131 (772h29m46.215629772s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235848449 (772h29m46.216201078s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.236290164 (772h29m46.216642812s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237033388 (772h29m46.217386026s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237491382 (772h29m46.217844022s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.238084906 (772h29m46.218437545s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.239186345 (772h29m46.219538984s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.240506522 (772h29m46.220859165s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.829743229 (772h29m46.810095872s) [upload=679912]: Next returned nil segment
Jan 6 19:58:06.829762199 (772h29m46.810114841s) [upload=679912]: Waiting for error group managing segments...
Jan 6 19:58:06.947371887 (772h29m46.927724525s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.002507087 (772h29m46.982859763s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.039550016 (772h29m47.019902653s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.073221490 (772h29m47.053574126s) [noise=true , address=77.90.31.18:4834 , piece_id=IXY6FIN7YKAX7HIE6YYF6C3CT6JQB7OCD5YQA2R3NXPV7ZSVNO5A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.121426552 (772h29m47.101779189s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.147306902 (772h29m47.127659539s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.155135219 (772h29m47.135487858s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.307741749 (772h29m47.288094384s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.440337127 (772h29m47.420689763s) [noise=true , address=45.91.116.13:4184 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.901506940 (772h29m47.88185958s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.932347050 (772h29m47.912699689s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.046430498 (772h29m48.026783134s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.063258213 (772h29m48.043610858s) [noise=true , address=145.79.227.14:4866 , piece_id=NMEQDY2COSGV6K4JIYXEYQ2A66BJRRCOAGCFS2URAOWXUB23QM4Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218417658 (772h29m48.198770297s) [noise=true , address=145.79.227.30:4789 , piece_id=UYSB4JQOVVVK6LYXE7A72MBUU4VGT7TAXXVGNTWY25JMO4OGK3WQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218446170 (772h29m48.198798807s) [seg_pos={0 0}, upload=679912]: Segment reached optimal threshold of 14 pieces.
Jan 6 19:58:08.218679983 (772h29m48.199032625s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection
Jan 6 19:58:08.218688530 (772h29m48.199041166s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218829688 (772h29m48.199182333s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:08.218854097 (772h29m48.19920674s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218942630 (772h29m48.199295275s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: read tcp 139.59.144.22:56426->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 19:58:08.218950936 (772h29m48.199303572s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219048646 (772h29m48.199401291s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection
Jan 6 19:58:08.219057565 (772h29m48.199410201s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219089322 (772h29m48.199441949s) [seg_pos={0 0}, upload=679912]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 19:58:08.219237200 (772h29m48.199589839s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:08.219244038 (772h29m48.199596674s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:08.352159986 (772h29m48.332512623s) [upload=679912]: Done uploading segments...
Jan 6 19:58:08.352210140 (772h29m48.332562776s) [upload=679912]: Upload finished. err: <nil>
Duration: 3.155794576s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
2.408110053s trace:
https://jaeger.dev.storj.io/trace/6e0ed516680ebec8
FAIL (2026-01-06 20:05:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A) context deadline exceeded
======LOGS======
Jan 6 19:58:17.403555994 (772h29m57.383908645s) [upload=679917]: Starting upload
Jan 6 19:58:17.403666206 (772h29m57.384018858s) [upload=679917]: Done starting upload
Jan 6 19:58:17.403842502 (772h29m57.38419514s) [upload=679917]: Uploading segments...
Jan 6 19:58:17.404166965 (772h29m57.384519606s) [upload=679917]: Got next segment. Inline: false
Jan 6 19:58:17.404223968 (772h29m57.38457663s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 19:58:17.404230594 (772h29m57.384583235s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 19:58:17.516709204 (772h29m57.49706185s) [seg_pos={0 0}, upload=679917]: Begin upload segment...
Jan 6 19:58:17.516795101 (772h29m57.497147747s) [seg_pos={0 0}, upload=679917]: Done begin upload segment.
Jan 6 19:58:17.516832647 (772h29m57.497185292s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.517030264 (772h29m57.497382912s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518092013 (772h29m57.498444671s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518958504 (772h29m57.49931115s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519239753 (772h29m57.4995924s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519449404 (772h29m57.499802051s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519955276 (772h29m57.500307924s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520193176 (772h29m57.500545822s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520785527 (772h29m57.501138173s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521208368 (772h29m57.501561018s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521330375 (772h29m57.501683021s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521967075 (772h29m57.502319718s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522287136 (772h29m57.502639784s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522546502 (772h29m57.502899148s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.523557905 (772h29m57.50391057s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.524586930 (772h29m57.504939577s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:18.166215094 (772h29m58.146567733s) [upload=679917]: Next returned nil segment
Jan 6 19:58:18.166228823 (772h29m58.14658146s) [upload=679917]: Waiting for error group managing segments...
Jan 6 19:58:18.302286606 (772h29m58.282639244s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.343646848 (772h29m58.323999484s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.357722828 (772h29m58.338075467s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.367003674 (772h29m58.347356311s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.376262781 (772h29m58.356615421s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.403506818 (772h29m58.383859465s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.465404723 (772h29m58.445757372s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.567776867 (772h29m58.548129504s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.599132864 (772h29m58.579485506s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.613759829 (772h29m58.594112475s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.664243429 (772h29m58.644596056s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.895647542 (772h29m58.876000181s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102892198 (772h29m59.08324489s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102980997 (772h29m59.083333641s) [seg_pos={0 0}, upload=679917]: Segment reached optimal threshold of 13 pieces.
Jan 6 19:58:19.103439612 (772h29m59.083792261s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.103456166 (772h29m59.083808814s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.103732510 (772h29m59.084085155s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection
Jan 6 19:58:19.103739973 (772h29m59.084092617s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104075008 (772h29m59.084427655s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.104083229 (772h29m59.08443587s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104103417 (772h29m59.084456044s) [seg_pos={0 0}, upload=679917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 19:58:19.104195361 (772h29m59.084548002s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:19.104203262 (772h29m59.08455589s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:19.248322143 (772h29m59.228674781s) [upload=679917]: Done uploading segments...
Jan 6 19:58:19.248344948 (772h29m59.228697584s) [upload=679917]: Upload finished. err: <nil>
Duration: 2.408154553s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
9.566202179s trace:
https://jaeger.dev.storj.io/trace/5cf4e6a4e2080b2d
FAIL (2026-01-06 20:04:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
4 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.233875032 (772h29m46.214227679s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234056119 (772h29m46.21440876s) [seg_pos={0 0}, upload=679912]: Done begin upload segment.
Jan 6 19:58:06.234140741 (772h29m46.214493379s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234498043 (772h29m46.214850694s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234792560 (772h29m46.215145197s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235003281 (772h29m46.215355922s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235277131 (772h29m46.215629772s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235848449 (772h29m46.216201078s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.236290164 (772h29m46.216642812s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237033388 (772h29m46.217386026s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237491382 (772h29m46.217844022s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.238084906 (772h29m46.218437545s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.239186345 (772h29m46.219538984s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.240506522 (772h29m46.220859165s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.829743229 (772h29m46.810095872s) [upload=679912]: Next returned nil segment
Jan 6 19:58:06.829762199 (772h29m46.810114841s) [upload=679912]: Waiting for error group managing segments...
Jan 6 19:58:06.947371887 (772h29m46.927724525s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.002507087 (772h29m46.982859763s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.039550016 (772h29m47.019902653s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.073221490 (772h29m47.053574126s) [noise=true , address=77.90.31.18:4834 , piece_id=IXY6FIN7YKAX7HIE6YYF6C3CT6JQB7OCD5YQA2R3NXPV7ZSVNO5A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.121426552 (772h29m47.101779189s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.147306902 (772h29m47.127659539s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.155135219 (772h29m47.135487858s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.307741749 (772h29m47.288094384s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.440337127 (772h29m47.420689763s) [noise=true , address=45.91.116.13:4184 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.901506940 (772h29m47.88185958s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.932347050 (772h29m47.912699689s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.046430498 (772h29m48.026783134s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.063258213 (772h29m48.043610858s) [noise=true , address=145.79.227.14:4866 , piece_id=NMEQDY2COSGV6K4JIYXEYQ2A66BJRRCOAGCFS2URAOWXUB23QM4Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218417658 (772h29m48.198770297s) [noise=true , address=145.79.227.30:4789 , piece_id=UYSB4JQOVVVK6LYXE7A72MBUU4VGT7TAXXVGNTWY25JMO4OGK3WQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218446170 (772h29m48.198798807s) [seg_pos={0 0}, upload=679912]: Segment reached optimal threshold of 14 pieces.
Jan 6 19:58:08.218679983 (772h29m48.199032625s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection
Jan 6 19:58:08.218688530 (772h29m48.199041166s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218829688 (772h29m48.199182333s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:08.218854097 (772h29m48.19920674s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218942630 (772h29m48.199295275s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: read tcp 139.59.144.22:56426->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 19:58:08.218950936 (772h29m48.199303572s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219048646 (772h29m48.199401291s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection
Jan 6 19:58:08.219057565 (772h29m48.199410201s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219089322 (772h29m48.199441949s) [seg_pos={0 0}, upload=679912]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 19:58:08.219237200 (772h29m48.199589839s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:08.219244038 (772h29m48.199596674s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:08.352159986 (772h29m48.332512623s) [upload=679912]: Done uploading segments...
Jan 6 19:58:08.352210140 (772h29m48.332562776s) [upload=679912]: Upload finished. err: <nil>
Duration: 9.566235187s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
2.323104785s trace:
https://jaeger.dev.storj.io/trace/6e0ed516680ebec8
FAIL (2026-01-06 20:04:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A) context deadline exceeded
======LOGS======
Jan 6 19:58:17.403555994 (772h29m57.383908645s) [upload=679917]: Starting upload
Jan 6 19:58:17.403666206 (772h29m57.384018858s) [upload=679917]: Done starting upload
Jan 6 19:58:17.403842502 (772h29m57.38419514s) [upload=679917]: Uploading segments...
Jan 6 19:58:17.404166965 (772h29m57.384519606s) [upload=679917]: Got next segment. Inline: false
Jan 6 19:58:17.404223968 (772h29m57.38457663s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 19:58:17.404230594 (772h29m57.384583235s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 19:58:17.516709204 (772h29m57.49706185s) [seg_pos={0 0}, upload=679917]: Begin upload segment...
Jan 6 19:58:17.516795101 (772h29m57.497147747s) [seg_pos={0 0}, upload=679917]: Done begin upload segment.
Jan 6 19:58:17.516832647 (772h29m57.497185292s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.517030264 (772h29m57.497382912s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518092013 (772h29m57.498444671s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518958504 (772h29m57.49931115s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519239753 (772h29m57.4995924s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519449404 (772h29m57.499802051s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519955276 (772h29m57.500307924s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520193176 (772h29m57.500545822s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520785527 (772h29m57.501138173s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521208368 (772h29m57.501561018s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521330375 (772h29m57.501683021s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521967075 (772h29m57.502319718s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522287136 (772h29m57.502639784s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522546502 (772h29m57.502899148s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.523557905 (772h29m57.50391057s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.524586930 (772h29m57.504939577s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:18.166215094 (772h29m58.146567733s) [upload=679917]: Next returned nil segment
Jan 6 19:58:18.166228823 (772h29m58.14658146s) [upload=679917]: Waiting for error group managing segments...
Jan 6 19:58:18.302286606 (772h29m58.282639244s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.343646848 (772h29m58.323999484s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.357722828 (772h29m58.338075467s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.367003674 (772h29m58.347356311s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.376262781 (772h29m58.356615421s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.403506818 (772h29m58.383859465s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.465404723 (772h29m58.445757372s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.567776867 (772h29m58.548129504s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.599132864 (772h29m58.579485506s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.613759829 (772h29m58.594112475s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.664243429 (772h29m58.644596056s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.895647542 (772h29m58.876000181s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102892198 (772h29m59.08324489s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102980997 (772h29m59.083333641s) [seg_pos={0 0}, upload=679917]: Segment reached optimal threshold of 13 pieces.
Jan 6 19:58:19.103439612 (772h29m59.083792261s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.103456166 (772h29m59.083808814s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.103732510 (772h29m59.084085155s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection
Jan 6 19:58:19.103739973 (772h29m59.084092617s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104075008 (772h29m59.084427655s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.104083229 (772h29m59.08443587s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104103417 (772h29m59.084456044s) [seg_pos={0 0}, upload=679917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 19:58:19.104195361 (772h29m59.084548002s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:19.104203262 (772h29m59.08455589s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:19.248322143 (772h29m59.228674781s) [upload=679917]: Done uploading segments...
Jan 6 19:58:19.248344948 (772h29m59.228697584s) [upload=679917]: Upload finished. err: <nil>
Duration: 2.323185208s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
12.69590664s trace:
https://jaeger.dev.storj.io/trace/5cf4e6a4e2080b2d
FAIL (2026-01-06 20:04:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
4 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.233875032 (772h29m46.214227679s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234056119 (772h29m46.21440876s) [seg_pos={0 0}, upload=679912]: Done begin upload segment.
Jan 6 19:58:06.234140741 (772h29m46.214493379s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234498043 (772h29m46.214850694s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234792560 (772h29m46.215145197s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235003281 (772h29m46.215355922s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235277131 (772h29m46.215629772s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235848449 (772h29m46.216201078s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.236290164 (772h29m46.216642812s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237033388 (772h29m46.217386026s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237491382 (772h29m46.217844022s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.238084906 (772h29m46.218437545s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.239186345 (772h29m46.219538984s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.240506522 (772h29m46.220859165s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.829743229 (772h29m46.810095872s) [upload=679912]: Next returned nil segment
Jan 6 19:58:06.829762199 (772h29m46.810114841s) [upload=679912]: Waiting for error group managing segments...
Jan 6 19:58:06.947371887 (772h29m46.927724525s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.002507087 (772h29m46.982859763s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.039550016 (772h29m47.019902653s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.073221490 (772h29m47.053574126s) [noise=true , address=77.90.31.18:4834 , piece_id=IXY6FIN7YKAX7HIE6YYF6C3CT6JQB7OCD5YQA2R3NXPV7ZSVNO5A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.121426552 (772h29m47.101779189s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.147306902 (772h29m47.127659539s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.155135219 (772h29m47.135487858s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.307741749 (772h29m47.288094384s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.440337127 (772h29m47.420689763s) [noise=true , address=45.91.116.13:4184 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.901506940 (772h29m47.88185958s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.932347050 (772h29m47.912699689s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.046430498 (772h29m48.026783134s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.063258213 (772h29m48.043610858s) [noise=true , address=145.79.227.14:4866 , piece_id=NMEQDY2COSGV6K4JIYXEYQ2A66BJRRCOAGCFS2URAOWXUB23QM4Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218417658 (772h29m48.198770297s) [noise=true , address=145.79.227.30:4789 , piece_id=UYSB4JQOVVVK6LYXE7A72MBUU4VGT7TAXXVGNTWY25JMO4OGK3WQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218446170 (772h29m48.198798807s) [seg_pos={0 0}, upload=679912]: Segment reached optimal threshold of 14 pieces.
Jan 6 19:58:08.218679983 (772h29m48.199032625s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection
Jan 6 19:58:08.218688530 (772h29m48.199041166s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218829688 (772h29m48.199182333s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:08.218854097 (772h29m48.19920674s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218942630 (772h29m48.199295275s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: read tcp 139.59.144.22:56426->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 19:58:08.218950936 (772h29m48.199303572s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219048646 (772h29m48.199401291s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection
Jan 6 19:58:08.219057565 (772h29m48.199410201s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219089322 (772h29m48.199441949s) [seg_pos={0 0}, upload=679912]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 19:58:08.219237200 (772h29m48.199589839s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:08.219244038 (772h29m48.199596674s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:08.352159986 (772h29m48.332512623s) [upload=679912]: Done uploading segments...
Jan 6 19:58:08.352210140 (772h29m48.332562776s) [upload=679912]: Upload finished. err: <nil>
Duration: 12.695931874s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
2.400517987s trace:
https://jaeger.dev.storj.io/trace/6e0ed516680ebec8
FAIL (2026-01-06 20:03:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A) context deadline exceeded
======LOGS======
Jan 6 19:58:17.403555994 (772h29m57.383908645s) [upload=679917]: Starting upload
Jan 6 19:58:17.403666206 (772h29m57.384018858s) [upload=679917]: Done starting upload
Jan 6 19:58:17.403842502 (772h29m57.38419514s) [upload=679917]: Uploading segments...
Jan 6 19:58:17.404166965 (772h29m57.384519606s) [upload=679917]: Got next segment. Inline: false
Jan 6 19:58:17.404223968 (772h29m57.38457663s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 19:58:17.404230594 (772h29m57.384583235s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 19:58:17.516709204 (772h29m57.49706185s) [seg_pos={0 0}, upload=679917]: Begin upload segment...
Jan 6 19:58:17.516795101 (772h29m57.497147747s) [seg_pos={0 0}, upload=679917]: Done begin upload segment.
Jan 6 19:58:17.516832647 (772h29m57.497185292s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.517030264 (772h29m57.497382912s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518092013 (772h29m57.498444671s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518958504 (772h29m57.49931115s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519239753 (772h29m57.4995924s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519449404 (772h29m57.499802051s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519955276 (772h29m57.500307924s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520193176 (772h29m57.500545822s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520785527 (772h29m57.501138173s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521208368 (772h29m57.501561018s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521330375 (772h29m57.501683021s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521967075 (772h29m57.502319718s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522287136 (772h29m57.502639784s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522546502 (772h29m57.502899148s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.523557905 (772h29m57.50391057s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.524586930 (772h29m57.504939577s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:18.166215094 (772h29m58.146567733s) [upload=679917]: Next returned nil segment
Jan 6 19:58:18.166228823 (772h29m58.14658146s) [upload=679917]: Waiting for error group managing segments...
Jan 6 19:58:18.302286606 (772h29m58.282639244s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.343646848 (772h29m58.323999484s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.357722828 (772h29m58.338075467s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.367003674 (772h29m58.347356311s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.376262781 (772h29m58.356615421s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.403506818 (772h29m58.383859465s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.465404723 (772h29m58.445757372s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.567776867 (772h29m58.548129504s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.599132864 (772h29m58.579485506s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.613759829 (772h29m58.594112475s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.664243429 (772h29m58.644596056s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.895647542 (772h29m58.876000181s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102892198 (772h29m59.08324489s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102980997 (772h29m59.083333641s) [seg_pos={0 0}, upload=679917]: Segment reached optimal threshold of 13 pieces.
Jan 6 19:58:19.103439612 (772h29m59.083792261s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.103456166 (772h29m59.083808814s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.103732510 (772h29m59.084085155s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection
Jan 6 19:58:19.103739973 (772h29m59.084092617s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104075008 (772h29m59.084427655s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.104083229 (772h29m59.08443587s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104103417 (772h29m59.084456044s) [seg_pos={0 0}, upload=679917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 19:58:19.104195361 (772h29m59.084548002s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:19.104203262 (772h29m59.08455589s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:19.248322143 (772h29m59.228674781s) [upload=679917]: Done uploading segments...
Jan 6 19:58:19.248344948 (772h29m59.228697584s) [upload=679917]: Upload finished. err: <nil>
Duration: 2.400549701s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
2.282150747s trace:
https://jaeger.dev.storj.io/trace/6e0ed516680ebec8
FAIL (2026-01-06 20:03:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A) context deadline exceeded
======LOGS======
Jan 6 19:58:17.403555994 (772h29m57.383908645s) [upload=679917]: Starting upload
Jan 6 19:58:17.403666206 (772h29m57.384018858s) [upload=679917]: Done starting upload
Jan 6 19:58:17.403842502 (772h29m57.38419514s) [upload=679917]: Uploading segments...
Jan 6 19:58:17.404166965 (772h29m57.384519606s) [upload=679917]: Got next segment. Inline: false
Jan 6 19:58:17.404223968 (772h29m57.38457663s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 19:58:17.404230594 (772h29m57.384583235s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 19:58:17.516709204 (772h29m57.49706185s) [seg_pos={0 0}, upload=679917]: Begin upload segment...
Jan 6 19:58:17.516795101 (772h29m57.497147747s) [seg_pos={0 0}, upload=679917]: Done begin upload segment.
Jan 6 19:58:17.516832647 (772h29m57.497185292s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.517030264 (772h29m57.497382912s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518092013 (772h29m57.498444671s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518958504 (772h29m57.49931115s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519239753 (772h29m57.4995924s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519449404 (772h29m57.499802051s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519955276 (772h29m57.500307924s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520193176 (772h29m57.500545822s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520785527 (772h29m57.501138173s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521208368 (772h29m57.501561018s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521330375 (772h29m57.501683021s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521967075 (772h29m57.502319718s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522287136 (772h29m57.502639784s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522546502 (772h29m57.502899148s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.523557905 (772h29m57.50391057s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.524586930 (772h29m57.504939577s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:18.166215094 (772h29m58.146567733s) [upload=679917]: Next returned nil segment
Jan 6 19:58:18.166228823 (772h29m58.14658146s) [upload=679917]: Waiting for error group managing segments...
Jan 6 19:58:18.302286606 (772h29m58.282639244s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.343646848 (772h29m58.323999484s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.357722828 (772h29m58.338075467s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.367003674 (772h29m58.347356311s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.376262781 (772h29m58.356615421s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.403506818 (772h29m58.383859465s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.465404723 (772h29m58.445757372s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.567776867 (772h29m58.548129504s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.599132864 (772h29m58.579485506s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.613759829 (772h29m58.594112475s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.664243429 (772h29m58.644596056s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.895647542 (772h29m58.876000181s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102892198 (772h29m59.08324489s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102980997 (772h29m59.083333641s) [seg_pos={0 0}, upload=679917]: Segment reached optimal threshold of 13 pieces.
Jan 6 19:58:19.103439612 (772h29m59.083792261s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.103456166 (772h29m59.083808814s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.103732510 (772h29m59.084085155s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection
Jan 6 19:58:19.103739973 (772h29m59.084092617s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104075008 (772h29m59.084427655s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.104083229 (772h29m59.08443587s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104103417 (772h29m59.084456044s) [seg_pos={0 0}, upload=679917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 19:58:19.104195361 (772h29m59.084548002s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:19.104203262 (772h29m59.08455589s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:19.248322143 (772h29m59.228674781s) [upload=679917]: Done uploading segments...
Jan 6 19:58:19.248344948 (772h29m59.228697584s) [upload=679917]: Upload finished. err: <nil>
Duration: 2.282176712s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
12.650820137s trace:
https://jaeger.dev.storj.io/trace/5cf4e6a4e2080b2d
FAIL (2026-01-06 20:03:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
4 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.233875032 (772h29m46.214227679s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234056119 (772h29m46.21440876s) [seg_pos={0 0}, upload=679912]: Done begin upload segment.
Jan 6 19:58:06.234140741 (772h29m46.214493379s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234498043 (772h29m46.214850694s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234792560 (772h29m46.215145197s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235003281 (772h29m46.215355922s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235277131 (772h29m46.215629772s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235848449 (772h29m46.216201078s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.236290164 (772h29m46.216642812s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237033388 (772h29m46.217386026s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237491382 (772h29m46.217844022s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.238084906 (772h29m46.218437545s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.239186345 (772h29m46.219538984s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.240506522 (772h29m46.220859165s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.829743229 (772h29m46.810095872s) [upload=679912]: Next returned nil segment
Jan 6 19:58:06.829762199 (772h29m46.810114841s) [upload=679912]: Waiting for error group managing segments...
Jan 6 19:58:06.947371887 (772h29m46.927724525s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.002507087 (772h29m46.982859763s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.039550016 (772h29m47.019902653s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.073221490 (772h29m47.053574126s) [noise=true , address=77.90.31.18:4834 , piece_id=IXY6FIN7YKAX7HIE6YYF6C3CT6JQB7OCD5YQA2R3NXPV7ZSVNO5A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.121426552 (772h29m47.101779189s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.147306902 (772h29m47.127659539s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.155135219 (772h29m47.135487858s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.307741749 (772h29m47.288094384s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.440337127 (772h29m47.420689763s) [noise=true , address=45.91.116.13:4184 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.901506940 (772h29m47.88185958s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.932347050 (772h29m47.912699689s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.046430498 (772h29m48.026783134s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.063258213 (772h29m48.043610858s) [noise=true , address=145.79.227.14:4866 , piece_id=NMEQDY2COSGV6K4JIYXEYQ2A66BJRRCOAGCFS2URAOWXUB23QM4Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218417658 (772h29m48.198770297s) [noise=true , address=145.79.227.30:4789 , piece_id=UYSB4JQOVVVK6LYXE7A72MBUU4VGT7TAXXVGNTWY25JMO4OGK3WQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218446170 (772h29m48.198798807s) [seg_pos={0 0}, upload=679912]: Segment reached optimal threshold of 14 pieces.
Jan 6 19:58:08.218679983 (772h29m48.199032625s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection
Jan 6 19:58:08.218688530 (772h29m48.199041166s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218829688 (772h29m48.199182333s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:08.218854097 (772h29m48.19920674s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218942630 (772h29m48.199295275s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: read tcp 139.59.144.22:56426->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 19:58:08.218950936 (772h29m48.199303572s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219048646 (772h29m48.199401291s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection
Jan 6 19:58:08.219057565 (772h29m48.199410201s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219089322 (772h29m48.199441949s) [seg_pos={0 0}, upload=679912]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 19:58:08.219237200 (772h29m48.199589839s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:08.219244038 (772h29m48.199596674s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:08.352159986 (772h29m48.332512623s) [upload=679912]: Done uploading segments...
Jan 6 19:58:08.352210140 (772h29m48.332562776s) [upload=679912]: Upload finished. err: <nil>
Duration: 12.650847116s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
14.991230909s trace:
https://jaeger.dev.storj.io/trace/6e0ed516680ebec8
FAIL (2026-01-06 20:02:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A) context deadline exceeded
======LOGS======
Jan 6 19:58:17.403555994 (772h29m57.383908645s) [upload=679917]: Starting upload
Jan 6 19:58:17.403666206 (772h29m57.384018858s) [upload=679917]: Done starting upload
Jan 6 19:58:17.403842502 (772h29m57.38419514s) [upload=679917]: Uploading segments...
Jan 6 19:58:17.404166965 (772h29m57.384519606s) [upload=679917]: Got next segment. Inline: false
Jan 6 19:58:17.404223968 (772h29m57.38457663s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 19:58:17.404230594 (772h29m57.384583235s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 19:58:17.516709204 (772h29m57.49706185s) [seg_pos={0 0}, upload=679917]: Begin upload segment...
Jan 6 19:58:17.516795101 (772h29m57.497147747s) [seg_pos={0 0}, upload=679917]: Done begin upload segment.
Jan 6 19:58:17.516832647 (772h29m57.497185292s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.517030264 (772h29m57.497382912s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518092013 (772h29m57.498444671s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518958504 (772h29m57.49931115s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519239753 (772h29m57.4995924s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519449404 (772h29m57.499802051s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519955276 (772h29m57.500307924s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520193176 (772h29m57.500545822s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520785527 (772h29m57.501138173s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521208368 (772h29m57.501561018s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521330375 (772h29m57.501683021s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521967075 (772h29m57.502319718s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522287136 (772h29m57.502639784s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522546502 (772h29m57.502899148s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.523557905 (772h29m57.50391057s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.524586930 (772h29m57.504939577s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:18.166215094 (772h29m58.146567733s) [upload=679917]: Next returned nil segment
Jan 6 19:58:18.166228823 (772h29m58.14658146s) [upload=679917]: Waiting for error group managing segments...
Jan 6 19:58:18.302286606 (772h29m58.282639244s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.343646848 (772h29m58.323999484s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.357722828 (772h29m58.338075467s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.367003674 (772h29m58.347356311s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.376262781 (772h29m58.356615421s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.403506818 (772h29m58.383859465s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.465404723 (772h29m58.445757372s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.567776867 (772h29m58.548129504s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.599132864 (772h29m58.579485506s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.613759829 (772h29m58.594112475s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.664243429 (772h29m58.644596056s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.895647542 (772h29m58.876000181s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102892198 (772h29m59.08324489s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102980997 (772h29m59.083333641s) [seg_pos={0 0}, upload=679917]: Segment reached optimal threshold of 13 pieces.
Jan 6 19:58:19.103439612 (772h29m59.083792261s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.103456166 (772h29m59.083808814s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.103732510 (772h29m59.084085155s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection
Jan 6 19:58:19.103739973 (772h29m59.084092617s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104075008 (772h29m59.084427655s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.104083229 (772h29m59.08443587s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104103417 (772h29m59.084456044s) [seg_pos={0 0}, upload=679917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 19:58:19.104195361 (772h29m59.084548002s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:19.104203262 (772h29m59.08455589s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:19.248322143 (772h29m59.228674781s) [upload=679917]: Done uploading segments...
Jan 6 19:58:19.248344948 (772h29m59.228697584s) [upload=679917]: Upload finished. err: <nil>
Duration: 14.99126368s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
3.327735238s trace:
https://jaeger.dev.storj.io/trace/5cf4e6a4e2080b2d
FAIL (2026-01-06 20:02:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
4 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.233875032 (772h29m46.214227679s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234056119 (772h29m46.21440876s) [seg_pos={0 0}, upload=679912]: Done begin upload segment.
Jan 6 19:58:06.234140741 (772h29m46.214493379s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234498043 (772h29m46.214850694s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234792560 (772h29m46.215145197s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235003281 (772h29m46.215355922s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235277131 (772h29m46.215629772s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235848449 (772h29m46.216201078s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.236290164 (772h29m46.216642812s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237033388 (772h29m46.217386026s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237491382 (772h29m46.217844022s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.238084906 (772h29m46.218437545s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.239186345 (772h29m46.219538984s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.240506522 (772h29m46.220859165s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.829743229 (772h29m46.810095872s) [upload=679912]: Next returned nil segment
Jan 6 19:58:06.829762199 (772h29m46.810114841s) [upload=679912]: Waiting for error group managing segments...
Jan 6 19:58:06.947371887 (772h29m46.927724525s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.002507087 (772h29m46.982859763s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.039550016 (772h29m47.019902653s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.073221490 (772h29m47.053574126s) [noise=true , address=77.90.31.18:4834 , piece_id=IXY6FIN7YKAX7HIE6YYF6C3CT6JQB7OCD5YQA2R3NXPV7ZSVNO5A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.121426552 (772h29m47.101779189s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.147306902 (772h29m47.127659539s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.155135219 (772h29m47.135487858s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.307741749 (772h29m47.288094384s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.440337127 (772h29m47.420689763s) [noise=true , address=45.91.116.13:4184 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.901506940 (772h29m47.88185958s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.932347050 (772h29m47.912699689s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.046430498 (772h29m48.026783134s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.063258213 (772h29m48.043610858s) [noise=true , address=145.79.227.14:4866 , piece_id=NMEQDY2COSGV6K4JIYXEYQ2A66BJRRCOAGCFS2URAOWXUB23QM4Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218417658 (772h29m48.198770297s) [noise=true , address=145.79.227.30:4789 , piece_id=UYSB4JQOVVVK6LYXE7A72MBUU4VGT7TAXXVGNTWY25JMO4OGK3WQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218446170 (772h29m48.198798807s) [seg_pos={0 0}, upload=679912]: Segment reached optimal threshold of 14 pieces.
Jan 6 19:58:08.218679983 (772h29m48.199032625s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection
Jan 6 19:58:08.218688530 (772h29m48.199041166s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218829688 (772h29m48.199182333s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:08.218854097 (772h29m48.19920674s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218942630 (772h29m48.199295275s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: read tcp 139.59.144.22:56426->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 19:58:08.218950936 (772h29m48.199303572s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219048646 (772h29m48.199401291s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection
Jan 6 19:58:08.219057565 (772h29m48.199410201s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219089322 (772h29m48.199441949s) [seg_pos={0 0}, upload=679912]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 19:58:08.219237200 (772h29m48.199589839s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:08.219244038 (772h29m48.199596674s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:08.352159986 (772h29m48.332512623s) [upload=679912]: Done uploading segments...
Jan 6 19:58:08.352210140 (772h29m48.332562776s) [upload=679912]: Upload finished. err: <nil>
Duration: 3.327811776s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
2.378185317s trace:
https://jaeger.dev.storj.io/trace/6e0ed516680ebec8
FAIL (2026-01-06 20:01:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A) context deadline exceeded
======LOGS======
Jan 6 19:58:17.403555994 (772h29m57.383908645s) [upload=679917]: Starting upload
Jan 6 19:58:17.403666206 (772h29m57.384018858s) [upload=679917]: Done starting upload
Jan 6 19:58:17.403842502 (772h29m57.38419514s) [upload=679917]: Uploading segments...
Jan 6 19:58:17.404166965 (772h29m57.384519606s) [upload=679917]: Got next segment. Inline: false
Jan 6 19:58:17.404223968 (772h29m57.38457663s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 19:58:17.404230594 (772h29m57.384583235s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 19:58:17.516709204 (772h29m57.49706185s) [seg_pos={0 0}, upload=679917]: Begin upload segment...
Jan 6 19:58:17.516795101 (772h29m57.497147747s) [seg_pos={0 0}, upload=679917]: Done begin upload segment.
Jan 6 19:58:17.516832647 (772h29m57.497185292s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.517030264 (772h29m57.497382912s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518092013 (772h29m57.498444671s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518958504 (772h29m57.49931115s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519239753 (772h29m57.4995924s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519449404 (772h29m57.499802051s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519955276 (772h29m57.500307924s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520193176 (772h29m57.500545822s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520785527 (772h29m57.501138173s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521208368 (772h29m57.501561018s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521330375 (772h29m57.501683021s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521967075 (772h29m57.502319718s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522287136 (772h29m57.502639784s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522546502 (772h29m57.502899148s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.523557905 (772h29m57.50391057s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.524586930 (772h29m57.504939577s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:18.166215094 (772h29m58.146567733s) [upload=679917]: Next returned nil segment
Jan 6 19:58:18.166228823 (772h29m58.14658146s) [upload=679917]: Waiting for error group managing segments...
Jan 6 19:58:18.302286606 (772h29m58.282639244s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.343646848 (772h29m58.323999484s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.357722828 (772h29m58.338075467s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.367003674 (772h29m58.347356311s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.376262781 (772h29m58.356615421s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.403506818 (772h29m58.383859465s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.465404723 (772h29m58.445757372s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.567776867 (772h29m58.548129504s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.599132864 (772h29m58.579485506s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.613759829 (772h29m58.594112475s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.664243429 (772h29m58.644596056s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.895647542 (772h29m58.876000181s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102892198 (772h29m59.08324489s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102980997 (772h29m59.083333641s) [seg_pos={0 0}, upload=679917]: Segment reached optimal threshold of 13 pieces.
Jan 6 19:58:19.103439612 (772h29m59.083792261s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.103456166 (772h29m59.083808814s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.103732510 (772h29m59.084085155s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection
Jan 6 19:58:19.103739973 (772h29m59.084092617s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104075008 (772h29m59.084427655s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.104083229 (772h29m59.08443587s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104103417 (772h29m59.084456044s) [seg_pos={0 0}, upload=679917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 19:58:19.104195361 (772h29m59.084548002s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:19.104203262 (772h29m59.08455589s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:19.248322143 (772h29m59.228674781s) [upload=679917]: Done uploading segments...
Jan 6 19:58:19.248344948 (772h29m59.228697584s) [upload=679917]: Upload finished. err: <nil>
Duration: 2.37821584s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
3.253453311s trace:
https://jaeger.dev.storj.io/trace/5cf4e6a4e2080b2d
FAIL (2026-01-06 20:01:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
4 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.233875032 (772h29m46.214227679s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234056119 (772h29m46.21440876s) [seg_pos={0 0}, upload=679912]: Done begin upload segment.
Jan 6 19:58:06.234140741 (772h29m46.214493379s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234498043 (772h29m46.214850694s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234792560 (772h29m46.215145197s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235003281 (772h29m46.215355922s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235277131 (772h29m46.215629772s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235848449 (772h29m46.216201078s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.236290164 (772h29m46.216642812s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237033388 (772h29m46.217386026s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237491382 (772h29m46.217844022s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.238084906 (772h29m46.218437545s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.239186345 (772h29m46.219538984s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.240506522 (772h29m46.220859165s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.829743229 (772h29m46.810095872s) [upload=679912]: Next returned nil segment
Jan 6 19:58:06.829762199 (772h29m46.810114841s) [upload=679912]: Waiting for error group managing segments...
Jan 6 19:58:06.947371887 (772h29m46.927724525s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.002507087 (772h29m46.982859763s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.039550016 (772h29m47.019902653s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.073221490 (772h29m47.053574126s) [noise=true , address=77.90.31.18:4834 , piece_id=IXY6FIN7YKAX7HIE6YYF6C3CT6JQB7OCD5YQA2R3NXPV7ZSVNO5A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.121426552 (772h29m47.101779189s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.147306902 (772h29m47.127659539s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.155135219 (772h29m47.135487858s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.307741749 (772h29m47.288094384s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.440337127 (772h29m47.420689763s) [noise=true , address=45.91.116.13:4184 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.901506940 (772h29m47.88185958s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.932347050 (772h29m47.912699689s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.046430498 (772h29m48.026783134s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.063258213 (772h29m48.043610858s) [noise=true , address=145.79.227.14:4866 , piece_id=NMEQDY2COSGV6K4JIYXEYQ2A66BJRRCOAGCFS2URAOWXUB23QM4Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218417658 (772h29m48.198770297s) [noise=true , address=145.79.227.30:4789 , piece_id=UYSB4JQOVVVK6LYXE7A72MBUU4VGT7TAXXVGNTWY25JMO4OGK3WQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218446170 (772h29m48.198798807s) [seg_pos={0 0}, upload=679912]: Segment reached optimal threshold of 14 pieces.
Jan 6 19:58:08.218679983 (772h29m48.199032625s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection
Jan 6 19:58:08.218688530 (772h29m48.199041166s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218829688 (772h29m48.199182333s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:08.218854097 (772h29m48.19920674s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218942630 (772h29m48.199295275s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: read tcp 139.59.144.22:56426->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 19:58:08.218950936 (772h29m48.199303572s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219048646 (772h29m48.199401291s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection
Jan 6 19:58:08.219057565 (772h29m48.199410201s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219089322 (772h29m48.199441949s) [seg_pos={0 0}, upload=679912]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 19:58:08.219237200 (772h29m48.199589839s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:08.219244038 (772h29m48.199596674s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:08.352159986 (772h29m48.332512623s) [upload=679912]: Done uploading segments...
Jan 6 19:58:08.352210140 (772h29m48.332562776s) [upload=679912]: Upload finished. err: <nil>
Duration: 3.253484019s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
2.452709771s trace:
https://jaeger.dev.storj.io/trace/6e0ed516680ebec8
FAIL (2026-01-06 20:00:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A) context deadline exceeded
======LOGS======
Jan 6 19:58:17.403555994 (772h29m57.383908645s) [upload=679917]: Starting upload
Jan 6 19:58:17.403666206 (772h29m57.384018858s) [upload=679917]: Done starting upload
Jan 6 19:58:17.403842502 (772h29m57.38419514s) [upload=679917]: Uploading segments...
Jan 6 19:58:17.404166965 (772h29m57.384519606s) [upload=679917]: Got next segment. Inline: false
Jan 6 19:58:17.404223968 (772h29m57.38457663s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 19:58:17.404230594 (772h29m57.384583235s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 19:58:17.516709204 (772h29m57.49706185s) [seg_pos={0 0}, upload=679917]: Begin upload segment...
Jan 6 19:58:17.516795101 (772h29m57.497147747s) [seg_pos={0 0}, upload=679917]: Done begin upload segment.
Jan 6 19:58:17.516832647 (772h29m57.497185292s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.517030264 (772h29m57.497382912s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518092013 (772h29m57.498444671s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518958504 (772h29m57.49931115s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519239753 (772h29m57.4995924s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519449404 (772h29m57.499802051s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519955276 (772h29m57.500307924s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520193176 (772h29m57.500545822s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520785527 (772h29m57.501138173s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521208368 (772h29m57.501561018s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521330375 (772h29m57.501683021s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521967075 (772h29m57.502319718s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522287136 (772h29m57.502639784s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522546502 (772h29m57.502899148s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.523557905 (772h29m57.50391057s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.524586930 (772h29m57.504939577s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:18.166215094 (772h29m58.146567733s) [upload=679917]: Next returned nil segment
Jan 6 19:58:18.166228823 (772h29m58.14658146s) [upload=679917]: Waiting for error group managing segments...
Jan 6 19:58:18.302286606 (772h29m58.282639244s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.343646848 (772h29m58.323999484s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.357722828 (772h29m58.338075467s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.367003674 (772h29m58.347356311s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.376262781 (772h29m58.356615421s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.403506818 (772h29m58.383859465s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.465404723 (772h29m58.445757372s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.567776867 (772h29m58.548129504s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.599132864 (772h29m58.579485506s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.613759829 (772h29m58.594112475s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.664243429 (772h29m58.644596056s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.895647542 (772h29m58.876000181s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102892198 (772h29m59.08324489s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102980997 (772h29m59.083333641s) [seg_pos={0 0}, upload=679917]: Segment reached optimal threshold of 13 pieces.
Jan 6 19:58:19.103439612 (772h29m59.083792261s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.103456166 (772h29m59.083808814s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.103732510 (772h29m59.084085155s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection
Jan 6 19:58:19.103739973 (772h29m59.084092617s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104075008 (772h29m59.084427655s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.104083229 (772h29m59.08443587s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104103417 (772h29m59.084456044s) [seg_pos={0 0}, upload=679917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 19:58:19.104195361 (772h29m59.084548002s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:19.104203262 (772h29m59.08455589s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:19.248322143 (772h29m59.228674781s) [upload=679917]: Done uploading segments...
Jan 6 19:58:19.248344948 (772h29m59.228697584s) [upload=679917]: Upload finished. err: <nil>
Duration: 2.452737592s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
2.315592493s trace:
https://jaeger.dev.storj.io/trace/6e0ed516680ebec8
FAIL (2026-01-06 20:00:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A) context deadline exceeded
======LOGS======
Jan 6 19:58:17.403555994 (772h29m57.383908645s) [upload=679917]: Starting upload
Jan 6 19:58:17.403666206 (772h29m57.384018858s) [upload=679917]: Done starting upload
Jan 6 19:58:17.403842502 (772h29m57.38419514s) [upload=679917]: Uploading segments...
Jan 6 19:58:17.404166965 (772h29m57.384519606s) [upload=679917]: Got next segment. Inline: false
Jan 6 19:58:17.404223968 (772h29m57.38457663s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 19:58:17.404230594 (772h29m57.384583235s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 19:58:17.516709204 (772h29m57.49706185s) [seg_pos={0 0}, upload=679917]: Begin upload segment...
Jan 6 19:58:17.516795101 (772h29m57.497147747s) [seg_pos={0 0}, upload=679917]: Done begin upload segment.
Jan 6 19:58:17.516832647 (772h29m57.497185292s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.517030264 (772h29m57.497382912s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518092013 (772h29m57.498444671s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518958504 (772h29m57.49931115s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519239753 (772h29m57.4995924s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519449404 (772h29m57.499802051s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519955276 (772h29m57.500307924s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520193176 (772h29m57.500545822s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520785527 (772h29m57.501138173s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521208368 (772h29m57.501561018s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521330375 (772h29m57.501683021s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521967075 (772h29m57.502319718s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522287136 (772h29m57.502639784s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522546502 (772h29m57.502899148s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.523557905 (772h29m57.50391057s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.524586930 (772h29m57.504939577s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:18.166215094 (772h29m58.146567733s) [upload=679917]: Next returned nil segment
Jan 6 19:58:18.166228823 (772h29m58.14658146s) [upload=679917]: Waiting for error group managing segments...
Jan 6 19:58:18.302286606 (772h29m58.282639244s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.343646848 (772h29m58.323999484s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.357722828 (772h29m58.338075467s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.367003674 (772h29m58.347356311s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.376262781 (772h29m58.356615421s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.403506818 (772h29m58.383859465s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.465404723 (772h29m58.445757372s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.567776867 (772h29m58.548129504s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.599132864 (772h29m58.579485506s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.613759829 (772h29m58.594112475s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.664243429 (772h29m58.644596056s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.895647542 (772h29m58.876000181s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102892198 (772h29m59.08324489s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102980997 (772h29m59.083333641s) [seg_pos={0 0}, upload=679917]: Segment reached optimal threshold of 13 pieces.
Jan 6 19:58:19.103439612 (772h29m59.083792261s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.103456166 (772h29m59.083808814s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.103732510 (772h29m59.084085155s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection
Jan 6 19:58:19.103739973 (772h29m59.084092617s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104075008 (772h29m59.084427655s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.104083229 (772h29m59.08443587s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104103417 (772h29m59.084456044s) [seg_pos={0 0}, upload=679917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 19:58:19.104195361 (772h29m59.084548002s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:19.104203262 (772h29m59.08455589s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:19.248322143 (772h29m59.228674781s) [upload=679917]: Done uploading segments...
Jan 6 19:58:19.248344948 (772h29m59.228697584s) [upload=679917]: Upload finished. err: <nil>
Duration: 2.315627871s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
12.725787452s trace:
https://jaeger.dev.storj.io/trace/5cf4e6a4e2080b2d
FAIL (2026-01-06 20:00:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
4 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.233875032 (772h29m46.214227679s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234056119 (772h29m46.21440876s) [seg_pos={0 0}, upload=679912]: Done begin upload segment.
Jan 6 19:58:06.234140741 (772h29m46.214493379s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234498043 (772h29m46.214850694s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234792560 (772h29m46.215145197s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235003281 (772h29m46.215355922s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235277131 (772h29m46.215629772s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235848449 (772h29m46.216201078s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.236290164 (772h29m46.216642812s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237033388 (772h29m46.217386026s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237491382 (772h29m46.217844022s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.238084906 (772h29m46.218437545s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.239186345 (772h29m46.219538984s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.240506522 (772h29m46.220859165s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.829743229 (772h29m46.810095872s) [upload=679912]: Next returned nil segment
Jan 6 19:58:06.829762199 (772h29m46.810114841s) [upload=679912]: Waiting for error group managing segments...
Jan 6 19:58:06.947371887 (772h29m46.927724525s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.002507087 (772h29m46.982859763s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.039550016 (772h29m47.019902653s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.073221490 (772h29m47.053574126s) [noise=true , address=77.90.31.18:4834 , piece_id=IXY6FIN7YKAX7HIE6YYF6C3CT6JQB7OCD5YQA2R3NXPV7ZSVNO5A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.121426552 (772h29m47.101779189s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.147306902 (772h29m47.127659539s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.155135219 (772h29m47.135487858s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.307741749 (772h29m47.288094384s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.440337127 (772h29m47.420689763s) [noise=true , address=45.91.116.13:4184 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.901506940 (772h29m47.88185958s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.932347050 (772h29m47.912699689s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.046430498 (772h29m48.026783134s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.063258213 (772h29m48.043610858s) [noise=true , address=145.79.227.14:4866 , piece_id=NMEQDY2COSGV6K4JIYXEYQ2A66BJRRCOAGCFS2URAOWXUB23QM4Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218417658 (772h29m48.198770297s) [noise=true , address=145.79.227.30:4789 , piece_id=UYSB4JQOVVVK6LYXE7A72MBUU4VGT7TAXXVGNTWY25JMO4OGK3WQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218446170 (772h29m48.198798807s) [seg_pos={0 0}, upload=679912]: Segment reached optimal threshold of 14 pieces.
Jan 6 19:58:08.218679983 (772h29m48.199032625s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection
Jan 6 19:58:08.218688530 (772h29m48.199041166s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218829688 (772h29m48.199182333s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:08.218854097 (772h29m48.19920674s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218942630 (772h29m48.199295275s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: read tcp 139.59.144.22:56426->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 19:58:08.218950936 (772h29m48.199303572s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219048646 (772h29m48.199401291s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection
Jan 6 19:58:08.219057565 (772h29m48.199410201s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219089322 (772h29m48.199441949s) [seg_pos={0 0}, upload=679912]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 19:58:08.219237200 (772h29m48.199589839s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:08.219244038 (772h29m48.199596674s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:08.352159986 (772h29m48.332512623s) [upload=679912]: Done uploading segments...
Jan 6 19:58:08.352210140 (772h29m48.332562776s) [upload=679912]: Upload finished. err: <nil>
Duration: 12.725965639s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
2.920935575s trace:
https://jaeger.dev.storj.io/trace/6e0ed516680ebec8
FAIL (2026-01-06 19:59:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A) context deadline exceeded
======LOGS======
Jan 6 19:58:17.403555994 (772h29m57.383908645s) [upload=679917]: Starting upload
Jan 6 19:58:17.403666206 (772h29m57.384018858s) [upload=679917]: Done starting upload
Jan 6 19:58:17.403842502 (772h29m57.38419514s) [upload=679917]: Uploading segments...
Jan 6 19:58:17.404166965 (772h29m57.384519606s) [upload=679917]: Got next segment. Inline: false
Jan 6 19:58:17.404223968 (772h29m57.38457663s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 19:58:17.404230594 (772h29m57.384583235s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 19:58:17.516709204 (772h29m57.49706185s) [seg_pos={0 0}, upload=679917]: Begin upload segment...
Jan 6 19:58:17.516795101 (772h29m57.497147747s) [seg_pos={0 0}, upload=679917]: Done begin upload segment.
Jan 6 19:58:17.516832647 (772h29m57.497185292s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.517030264 (772h29m57.497382912s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518092013 (772h29m57.498444671s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518958504 (772h29m57.49931115s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519239753 (772h29m57.4995924s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519449404 (772h29m57.499802051s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519955276 (772h29m57.500307924s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520193176 (772h29m57.500545822s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520785527 (772h29m57.501138173s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521208368 (772h29m57.501561018s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521330375 (772h29m57.501683021s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521967075 (772h29m57.502319718s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522287136 (772h29m57.502639784s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522546502 (772h29m57.502899148s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.523557905 (772h29m57.50391057s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.524586930 (772h29m57.504939577s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:18.166215094 (772h29m58.146567733s) [upload=679917]: Next returned nil segment
Jan 6 19:58:18.166228823 (772h29m58.14658146s) [upload=679917]: Waiting for error group managing segments...
Jan 6 19:58:18.302286606 (772h29m58.282639244s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.343646848 (772h29m58.323999484s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.357722828 (772h29m58.338075467s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.367003674 (772h29m58.347356311s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.376262781 (772h29m58.356615421s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.403506818 (772h29m58.383859465s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.465404723 (772h29m58.445757372s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.567776867 (772h29m58.548129504s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.599132864 (772h29m58.579485506s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.613759829 (772h29m58.594112475s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.664243429 (772h29m58.644596056s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.895647542 (772h29m58.876000181s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102892198 (772h29m59.08324489s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102980997 (772h29m59.083333641s) [seg_pos={0 0}, upload=679917]: Segment reached optimal threshold of 13 pieces.
Jan 6 19:58:19.103439612 (772h29m59.083792261s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.103456166 (772h29m59.083808814s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.103732510 (772h29m59.084085155s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection
Jan 6 19:58:19.103739973 (772h29m59.084092617s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104075008 (772h29m59.084427655s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.104083229 (772h29m59.08443587s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104103417 (772h29m59.084456044s) [seg_pos={0 0}, upload=679917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 19:58:19.104195361 (772h29m59.084548002s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:19.104203262 (772h29m59.08455589s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:19.248322143 (772h29m59.228674781s) [upload=679917]: Done uploading segments...
Jan 6 19:58:19.248344948 (772h29m59.228697584s) [upload=679917]: Upload finished. err: <nil>
Duration: 2.920963756s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
2.285152389s trace:
https://jaeger.dev.storj.io/trace/6e0ed516680ebec8
FAIL (2026-01-06 19:59:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A) context deadline exceeded
======LOGS======
Jan 6 19:58:17.403555994 (772h29m57.383908645s) [upload=679917]: Starting upload
Jan 6 19:58:17.403666206 (772h29m57.384018858s) [upload=679917]: Done starting upload
Jan 6 19:58:17.403842502 (772h29m57.38419514s) [upload=679917]: Uploading segments...
Jan 6 19:58:17.404166965 (772h29m57.384519606s) [upload=679917]: Got next segment. Inline: false
Jan 6 19:58:17.404223968 (772h29m57.38457663s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 19:58:17.404230594 (772h29m57.384583235s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 19:58:17.516709204 (772h29m57.49706185s) [seg_pos={0 0}, upload=679917]: Begin upload segment...
Jan 6 19:58:17.516795101 (772h29m57.497147747s) [seg_pos={0 0}, upload=679917]: Done begin upload segment.
Jan 6 19:58:17.516832647 (772h29m57.497185292s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.517030264 (772h29m57.497382912s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518092013 (772h29m57.498444671s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518958504 (772h29m57.49931115s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519239753 (772h29m57.4995924s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519449404 (772h29m57.499802051s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519955276 (772h29m57.500307924s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520193176 (772h29m57.500545822s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520785527 (772h29m57.501138173s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521208368 (772h29m57.501561018s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521330375 (772h29m57.501683021s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521967075 (772h29m57.502319718s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522287136 (772h29m57.502639784s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522546502 (772h29m57.502899148s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.523557905 (772h29m57.50391057s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.524586930 (772h29m57.504939577s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:18.166215094 (772h29m58.146567733s) [upload=679917]: Next returned nil segment
Jan 6 19:58:18.166228823 (772h29m58.14658146s) [upload=679917]: Waiting for error group managing segments...
Jan 6 19:58:18.302286606 (772h29m58.282639244s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.343646848 (772h29m58.323999484s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.357722828 (772h29m58.338075467s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.367003674 (772h29m58.347356311s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.376262781 (772h29m58.356615421s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.403506818 (772h29m58.383859465s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.465404723 (772h29m58.445757372s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.567776867 (772h29m58.548129504s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.599132864 (772h29m58.579485506s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.613759829 (772h29m58.594112475s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.664243429 (772h29m58.644596056s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.895647542 (772h29m58.876000181s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102892198 (772h29m59.08324489s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102980997 (772h29m59.083333641s) [seg_pos={0 0}, upload=679917]: Segment reached optimal threshold of 13 pieces.
Jan 6 19:58:19.103439612 (772h29m59.083792261s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.103456166 (772h29m59.083808814s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.103732510 (772h29m59.084085155s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection
Jan 6 19:58:19.103739973 (772h29m59.084092617s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104075008 (772h29m59.084427655s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.104083229 (772h29m59.08443587s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104103417 (772h29m59.084456044s) [seg_pos={0 0}, upload=679917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 19:58:19.104195361 (772h29m59.084548002s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:19.104203262 (772h29m59.08455589s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:19.248322143 (772h29m59.228674781s) [upload=679917]: Done uploading segments...
Jan 6 19:58:19.248344948 (772h29m59.228697584s) [upload=679917]: Upload finished. err: <nil>
Duration: 2.285183035s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
3.241915062s trace:
https://jaeger.dev.storj.io/trace/5cf4e6a4e2080b2d
FAIL (2026-01-06 19:59:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
4 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.233875032 (772h29m46.214227679s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234056119 (772h29m46.21440876s) [seg_pos={0 0}, upload=679912]: Done begin upload segment.
Jan 6 19:58:06.234140741 (772h29m46.214493379s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234498043 (772h29m46.214850694s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234792560 (772h29m46.215145197s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235003281 (772h29m46.215355922s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235277131 (772h29m46.215629772s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235848449 (772h29m46.216201078s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.236290164 (772h29m46.216642812s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237033388 (772h29m46.217386026s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237491382 (772h29m46.217844022s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.238084906 (772h29m46.218437545s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.239186345 (772h29m46.219538984s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.240506522 (772h29m46.220859165s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.829743229 (772h29m46.810095872s) [upload=679912]: Next returned nil segment
Jan 6 19:58:06.829762199 (772h29m46.810114841s) [upload=679912]: Waiting for error group managing segments...
Jan 6 19:58:06.947371887 (772h29m46.927724525s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.002507087 (772h29m46.982859763s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.039550016 (772h29m47.019902653s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.073221490 (772h29m47.053574126s) [noise=true , address=77.90.31.18:4834 , piece_id=IXY6FIN7YKAX7HIE6YYF6C3CT6JQB7OCD5YQA2R3NXPV7ZSVNO5A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.121426552 (772h29m47.101779189s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.147306902 (772h29m47.127659539s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.155135219 (772h29m47.135487858s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.307741749 (772h29m47.288094384s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.440337127 (772h29m47.420689763s) [noise=true , address=45.91.116.13:4184 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.901506940 (772h29m47.88185958s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.932347050 (772h29m47.912699689s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.046430498 (772h29m48.026783134s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.063258213 (772h29m48.043610858s) [noise=true , address=145.79.227.14:4866 , piece_id=NMEQDY2COSGV6K4JIYXEYQ2A66BJRRCOAGCFS2URAOWXUB23QM4Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218417658 (772h29m48.198770297s) [noise=true , address=145.79.227.30:4789 , piece_id=UYSB4JQOVVVK6LYXE7A72MBUU4VGT7TAXXVGNTWY25JMO4OGK3WQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218446170 (772h29m48.198798807s) [seg_pos={0 0}, upload=679912]: Segment reached optimal threshold of 14 pieces.
Jan 6 19:58:08.218679983 (772h29m48.199032625s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection
Jan 6 19:58:08.218688530 (772h29m48.199041166s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218829688 (772h29m48.199182333s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:08.218854097 (772h29m48.19920674s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218942630 (772h29m48.199295275s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: read tcp 139.59.144.22:56426->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 19:58:08.218950936 (772h29m48.199303572s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219048646 (772h29m48.199401291s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection
Jan 6 19:58:08.219057565 (772h29m48.199410201s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219089322 (772h29m48.199441949s) [seg_pos={0 0}, upload=679912]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 19:58:08.219237200 (772h29m48.199589839s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:08.219244038 (772h29m48.199596674s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:08.352159986 (772h29m48.332512623s) [upload=679912]: Done uploading segments...
Jan 6 19:58:08.352210140 (772h29m48.332562776s) [upload=679912]: Upload finished. err: <nil>
Duration: 3.241945101s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
14.494466147s trace:
https://jaeger.dev.storj.io/trace/5cf4e6a4e2080b2d
FAIL (2026-01-06 19:59:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
4 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.233875032 (772h29m46.214227679s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234056119 (772h29m46.21440876s) [seg_pos={0 0}, upload=679912]: Done begin upload segment.
Jan 6 19:58:06.234140741 (772h29m46.214493379s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234498043 (772h29m46.214850694s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234792560 (772h29m46.215145197s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235003281 (772h29m46.215355922s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235277131 (772h29m46.215629772s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235848449 (772h29m46.216201078s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.236290164 (772h29m46.216642812s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237033388 (772h29m46.217386026s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237491382 (772h29m46.217844022s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.238084906 (772h29m46.218437545s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.239186345 (772h29m46.219538984s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.240506522 (772h29m46.220859165s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.829743229 (772h29m46.810095872s) [upload=679912]: Next returned nil segment
Jan 6 19:58:06.829762199 (772h29m46.810114841s) [upload=679912]: Waiting for error group managing segments...
Jan 6 19:58:06.947371887 (772h29m46.927724525s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.002507087 (772h29m46.982859763s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.039550016 (772h29m47.019902653s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.073221490 (772h29m47.053574126s) [noise=true , address=77.90.31.18:4834 , piece_id=IXY6FIN7YKAX7HIE6YYF6C3CT6JQB7OCD5YQA2R3NXPV7ZSVNO5A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.121426552 (772h29m47.101779189s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.147306902 (772h29m47.127659539s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.155135219 (772h29m47.135487858s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.307741749 (772h29m47.288094384s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.440337127 (772h29m47.420689763s) [noise=true , address=45.91.116.13:4184 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.901506940 (772h29m47.88185958s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.932347050 (772h29m47.912699689s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.046430498 (772h29m48.026783134s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.063258213 (772h29m48.043610858s) [noise=true , address=145.79.227.14:4866 , piece_id=NMEQDY2COSGV6K4JIYXEYQ2A66BJRRCOAGCFS2URAOWXUB23QM4Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218417658 (772h29m48.198770297s) [noise=true , address=145.79.227.30:4789 , piece_id=UYSB4JQOVVVK6LYXE7A72MBUU4VGT7TAXXVGNTWY25JMO4OGK3WQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218446170 (772h29m48.198798807s) [seg_pos={0 0}, upload=679912]: Segment reached optimal threshold of 14 pieces.
Jan 6 19:58:08.218679983 (772h29m48.199032625s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection
Jan 6 19:58:08.218688530 (772h29m48.199041166s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218829688 (772h29m48.199182333s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:08.218854097 (772h29m48.19920674s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218942630 (772h29m48.199295275s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: read tcp 139.59.144.22:56426->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 19:58:08.218950936 (772h29m48.199303572s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219048646 (772h29m48.199401291s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection
Jan 6 19:58:08.219057565 (772h29m48.199410201s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219089322 (772h29m48.199441949s) [seg_pos={0 0}, upload=679912]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 19:58:08.219237200 (772h29m48.199589839s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:08.219244038 (772h29m48.199596674s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:08.352159986 (772h29m48.332512623s) [upload=679912]: Done uploading segments...
Jan 6 19:58:08.352210140 (772h29m48.332562776s) [upload=679912]: Upload finished. err: <nil>
Duration: 14.494494621s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.005326183s trace:
https://jaeger.dev.storj.io/trace/6e0ed516680ebec8
FAIL (2026-01-06 19:58:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, Piece ID: 4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q) context deadline exceeded; piecestore: (Node ID: 121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b, Piece ID: HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A) context deadline exceeded
======LOGS======
Jan 6 19:58:17.403555994 (772h29m57.383908645s) [upload=679917]: Starting upload
Jan 6 19:58:17.403666206 (772h29m57.384018858s) [upload=679917]: Done starting upload
Jan 6 19:58:17.403842502 (772h29m57.38419514s) [upload=679917]: Uploading segments...
Jan 6 19:58:17.404166965 (772h29m57.384519606s) [upload=679917]: Got next segment. Inline: false
Jan 6 19:58:17.404223968 (772h29m57.38457663s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 6 19:58:17.404230594 (772h29m57.384583235s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 6 19:58:17.516709204 (772h29m57.49706185s) [seg_pos={0 0}, upload=679917]: Begin upload segment...
Jan 6 19:58:17.516795101 (772h29m57.497147747s) [seg_pos={0 0}, upload=679917]: Done begin upload segment.
Jan 6 19:58:17.516832647 (772h29m57.497185292s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.517030264 (772h29m57.497382912s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518092013 (772h29m57.498444671s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.518958504 (772h29m57.49931115s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519239753 (772h29m57.4995924s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519449404 (772h29m57.499802051s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.519955276 (772h29m57.500307924s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520193176 (772h29m57.500545822s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.520785527 (772h29m57.501138173s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521208368 (772h29m57.501561018s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521330375 (772h29m57.501683021s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.521967075 (772h29m57.502319718s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522287136 (772h29m57.502639784s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.522546502 (772h29m57.502899148s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.523557905 (772h29m57.50391057s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:17.524586930 (772h29m57.504939577s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Uploading piece...
Jan 6 19:58:18.166215094 (772h29m58.146567733s) [upload=679917]: Next returned nil segment
Jan 6 19:58:18.166228823 (772h29m58.14658146s) [upload=679917]: Waiting for error group managing segments...
Jan 6 19:58:18.302286606 (772h29m58.282639244s) [noise=true , address=77.90.31.11:4554 , piece_id=APVBHYAJTLHMGP43CYZH5GJI2WLU4B75VNOM23HAJDZXQKN52RBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.343646848 (772h29m58.323999484s) [noise=true , address=145.79.227.7:4294 , piece_id=4ASODFJ6D6DV7MOMS633CIPEXNZRTSGGMCNPXIW6EQYOYRZ7YB7Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.357722828 (772h29m58.338075467s) [noise=true , address=145.79.227.3:4149 , piece_id=WQTF2UFKP6F2WOCQRQRDOUGO2CKA5XM75RGMDGHLJXOPT5QE5GXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.367003674 (772h29m58.347356311s) [noise=true , address=145.79.227.21:4474 , piece_id=SRH43WGKHL4P6DSK6IEMIXCFWLGQB3LUB446TW5RSYPXVFHPGD4A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.376262781 (772h29m58.356615421s) [noise=true , address=145.79.227.51:4411 , piece_id=T3V2KXXU5RWNEL5PKZOLE52HWQ63EYVQEG5DIPQOKIPG4NB572EA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.403506818 (772h29m58.383859465s) [noise=true , address=45.91.116.13:4155 , piece_id=PSTDPHBFND4FY5QX2W6UEWRWWQNQKFDOI5EY5U3SV6PXYYIDINPA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.465404723 (772h29m58.445757372s) [noise=true , address=45.91.116.14:4190 , piece_id=YCIXPMM5W2S5MB7JMRZ4C6X52TTHKFU3JOR3GNKX47HMS2CLJPOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.567776867 (772h29m58.548129504s) [noise=true , address=145.79.227.50:4374 , piece_id=F7OF7WGKUV45AYBQV7VD4JWKAECXHXO25EMZFPTME7HO7L4ZPDBQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.599132864 (772h29m58.579485506s) [noise=true , address=145.79.227.22:4511 , piece_id=PPB74G37XMMLG6743DIU2ASUCHH4C3YFPTIKTDFTI6MOYVUJAWXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.613759829 (772h29m58.594112475s) [noise=true , address=145.79.227.39:4013 , piece_id=HMZ7HBXLP5WWXN6UATBJ5MYTLN3X6LE2O24UWSODROYXNKPAAS2A, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.664243429 (772h29m58.644596056s) [noise=true , address=145.79.227.8:4331 , piece_id=GOB5HGXBW3MGWNMITJWEJVEPS4LKBXUJCVJ3YEUEVORZIJ2CYS2Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:18.895647542 (772h29m58.876000181s) [noise=true , address=145.79.227.38:4978 , piece_id=R2ROZLP2S2SSQWILCC6ZKDBFFAEZRB326G4RIDTECJANC4HBMFXA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102892198 (772h29m59.08324489s) [noise=true , address=77.90.31.15:4697 , piece_id=ORJFXZE4C5N3HYURWP7ES6RICDAQOCFWTQQGAQL4CIJKETA4T4JA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: <nil>
Jan 6 19:58:19.102980997 (772h29m59.083333641s) [seg_pos={0 0}, upload=679917]: Segment reached optimal threshold of 13 pieces.
Jan 6 19:58:19.103439612 (772h29m59.083792261s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.103456166 (772h29m59.083808814s) [noise=true , address=145.79.227.41:4086 , piece_id=JGT2SXDDBVFZY7ITAWWOUVKOJNRWPTKBOQCOTVETV6PLENMM7WOQ, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.103732510 (772h29m59.084085155s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:50926->77.90.31.18:4805: use of closed network connection
Jan 6 19:58:19.103739973 (772h29m59.084092617s) [noise=true , address=77.90.31.18:4805 , piece_id=YURVZENWQ7A4XXCHBBUW2EIB33MJHTREXENTIXEAM7ODIXP5DT4Q, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104075008 (772h29m59.084427655s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:19.104083229 (772h29m59.08443587s) [noise=true , address=145.79.227.35:4482 , piece_id=O5SVWJCLHAMEKCNVTZHCQQS32N6LDHGUWZ2J4WPIRLNW47RXMKBA, seg_pos={0 0}, upload=679917]: Long tail cancellation due to optimal threshold
Jan 6 19:58:19.104103417 (772h29m59.084456044s) [seg_pos={0 0}, upload=679917]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 6 19:58:19.104195361 (772h29m59.084548002s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:19.104203262 (772h29m59.08455589s) [upload=679917]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:19.248322143 (772h29m59.228674781s) [upload=679917]: Done uploading segments...
Jan 6 19:58:19.248344948 (772h29m59.228697584s) [upload=679917]: Upload finished. err: <nil>
Duration: 30.005353632s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
39.936141994s trace:
https://jaeger.dev.storj.io/trace/5cf4e6a4e2080b2d
FAIL (2026-01-06 19:58:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
4 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.233875032 (772h29m46.214227679s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234056119 (772h29m46.21440876s) [seg_pos={0 0}, upload=679912]: Done begin upload segment.
Jan 6 19:58:06.234140741 (772h29m46.214493379s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234498043 (772h29m46.214850694s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.234792560 (772h29m46.215145197s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235003281 (772h29m46.215355922s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235277131 (772h29m46.215629772s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.235848449 (772h29m46.216201078s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.236290164 (772h29m46.216642812s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237033388 (772h29m46.217386026s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.237491382 (772h29m46.217844022s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.238084906 (772h29m46.218437545s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.239186345 (772h29m46.219538984s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.240506522 (772h29m46.220859165s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Uploading piece...
Jan 6 19:58:06.829743229 (772h29m46.810095872s) [upload=679912]: Next returned nil segment
Jan 6 19:58:06.829762199 (772h29m46.810114841s) [upload=679912]: Waiting for error group managing segments...
Jan 6 19:58:06.947371887 (772h29m46.927724525s) [noise=true , address=145.79.227.32:4847 , piece_id=6U4VE2TTWB6DHTV2BHGWHPCZV7U64CS6RJULYUOKG2VTC27VCTRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.002507087 (772h29m46.982859763s) [noise=true , address=45.91.116.9:4266 , piece_id=GRLWIMJIAAUAJO3RISE3JWGJYCCQYWF7JRZFZR3SS46IUET67GPA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.039550016 (772h29m47.019902653s) [noise=true , address=145.79.227.32:4866 , piece_id=4DYLXPB62ILGSR7D3UD2AGN6QRE3TSXG423GEKDP4ZRQ4GAHC43Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.073221490 (772h29m47.053574126s) [noise=true , address=77.90.31.18:4834 , piece_id=IXY6FIN7YKAX7HIE6YYF6C3CT6JQB7OCD5YQA2R3NXPV7ZSVNO5A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.121426552 (772h29m47.101779189s) [noise=true , address=5.231.73.3:4020 , piece_id=XK5TJZYM3XT6RO3ZVA47TWZ5XY3HMUG77I6KIQKVVRRZ4YDFLQUQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.147306902 (772h29m47.127659539s) [noise=true , address=77.90.31.12:4620 , piece_id=6XLSTU3WU37AUBBK5BHHUYRNPSZ3XZFMMRNYPSHNDH5YSWJZXK3A, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.155135219 (772h29m47.135487858s) [noise=true , address=45.91.116.14:4202 , piece_id=HS3UHWQQ6APNCIVTI65RJDM2YLSPBR3EXEKBOII2EGB7J72SNKRA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.307741749 (772h29m47.288094384s) [noise=true , address=77.90.31.19:4874 , piece_id=ADOYQBYXPJP4GBZA5D6IB72OV5KSE6ESPPMH7C6LWFJIJRIL42FA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.440337127 (772h29m47.420689763s) [noise=true , address=45.91.116.13:4184 , piece_id=TYYXY6NHIXCO7OSL2RD3OKGOKU4MFRGT2BPCHEFYQ6B52VCFY2XQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.901506940 (772h29m47.88185958s) [noise=true , address=145.79.227.3:4170 , piece_id=SMIF5I6K5PZFLIKQ22Y7JWIVHT6SWVPLQALRIH5R6L4FVHVC7NGQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:07.932347050 (772h29m47.912699689s) [noise=true , address=145.79.227.29:4969 , piece_id=DNV6MFNNMQL5ZVQK2AIGYIAS6E5THGUDI4X4DLPTMJIH7YCPMPPQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.046430498 (772h29m48.026783134s) [noise=true , address=145.79.227.43:4168 , piece_id=NOFNWYEMG5BP622VSJGSR65FY264DRNWGSLI6YUXV6FTW7DXOOCA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.063258213 (772h29m48.043610858s) [noise=true , address=145.79.227.14:4866 , piece_id=NMEQDY2COSGV6K4JIYXEYQ2A66BJRRCOAGCFS2URAOWXUB23QM4Q, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218417658 (772h29m48.198770297s) [noise=true , address=145.79.227.30:4789 , piece_id=UYSB4JQOVVVK6LYXE7A72MBUU4VGT7TAXXVGNTWY25JMO4OGK3WQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: <nil>
Jan 6 19:58:08.218446170 (772h29m48.198798807s) [seg_pos={0 0}, upload=679912]: Segment reached optimal threshold of 14 pieces.
Jan 6 19:58:08.218679983 (772h29m48.199032625s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BdVmmJ1L1sEMcHXTAdXBXU6dmdhUE9VT74jCNdZj5bMqJWxqp): protocol: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33372->145.79.227.17:4004: use of closed network connection
Jan 6 19:58:08.218688530 (772h29m48.199041166s) [noise=true , address=145.79.227.17:4004 , piece_id=QTPQKULKLEVKTBGB77DOQ47U4KL5IA2ND5PM4TTUBXVUXVVXXKAA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218829688 (772h29m48.199182333s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16BjXZvkSyZbchwbLZpL5b4sJDvoUmaxNXsMu9j2Bhcx4WBrjJ): protocol: expected piece hash; context canceled; EOF
Jan 6 19:58:08.218854097 (772h29m48.19920674s) [noise=true , address=193.149.164.143:4697 , piece_id=W5TECRY7CFBI2K6UYXYHNJP3YY3KCHOG4Q7VJAN7GC3EJ5FGJEZA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.218942630 (772h29m48.199295275s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rZhbymrDS6ahXLEqYRM64gcqRvG8CEnc8iGCTEhgXrpvRaQrp): protocol: read tcp 139.59.144.22:56426->145.79.227.47:4285: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 6 19:58:08.218950936 (772h29m48.199303572s) [noise=true , address=145.79.227.47:4285 , piece_id=VUAB2QIWWFUJGCCELFDEKTMYG7YKL64I2AYGQOSA7X2EM6WKXCNA, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219048646 (772h29m48.199401291s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F63eNzovFha5jv7syk3XYzn1tqDgDwm68HpHd56cNJKAsFcoe): protocol: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:44910->145.79.227.40:4069: use of closed network connection
Jan 6 19:58:08.219057565 (772h29m48.199410201s) [noise=true , address=145.79.227.40:4069 , piece_id=7RMXA4M66ZXOHHKR5FCSHBVFWUZNE3G2JVT2QFIOLOXHJ2GT6OHQ, seg_pos={0 0}, upload=679912]: Long tail cancellation due to optimal threshold
Jan 6 19:58:08.219089322 (772h29m48.199441949s) [seg_pos={0 0}, upload=679912]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 6 19:58:08.219237200 (772h29m48.199589839s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 6 19:58:08.219244038 (772h29m48.199596674s) [upload=679912]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 6 19:58:08.352159986 (772h29m48.332512623s) [upload=679912]: Done uploading segments...
Jan 6 19:58:08.352210140 (772h29m48.332562776s) [upload=679912]: Upload finished. err: <nil>
Duration: 39.936189775s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
2.190796956s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 19:05:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.190866889s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.X..........................................................] duration:
2.198880949s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 19:05:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.198916945s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..X.........................................................] duration:
1.613281279s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 19:03:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.613316207s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...X........................................................] duration:
1.96341645s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 19:03:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.963441507s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....X.......................................................] duration:
2.212349638s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 19:01:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.212439817s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....X......................................................] duration:
1.813016574s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 19:01:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.813056869s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......X.....................................................] duration:
1.661765966s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 19:01:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.661860453s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......X....................................................] duration:
1.550477338s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 19:00:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.550517674s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........X...................................................] duration:
1.558013792s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:59:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.55803805s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........X..................................................] duration:
1.644561548s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:58:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.644589864s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........X.................................................] duration:
1.650989596s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:57:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.651018735s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........X................................................] duration:
1.610701168s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:56:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.610777237s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [............X...............................................] duration:
1.589277681s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:56:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.589301915s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............X..............................................] duration:
1.55101872s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:55:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.551141711s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............X.............................................] duration:
1.556390066s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:54:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.556418102s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...............X............................................] duration:
1.787631233s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:53:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.787661769s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................X...........................................] duration:
1.577330759s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:53:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.577409712s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................X..........................................] duration:
1.654892335s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:52:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.654935164s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..................X.........................................] duration:
1.814195897s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:52:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.814235605s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...................X........................................] duration:
1.599717875s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:51:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.599757667s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....................X.......................................] duration:
1.566279932s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:50:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.56647221s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....................X......................................] duration:
1.747363905s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:50:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.747387994s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......................X.....................................] duration:
1.700146616s trace:
https://jaeger.dev.storj.io/trace/5e73b7fd558c263c
FAIL (2026-01-06 18:49:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.700182344s
core
uplink
us-select-1
us1-placement-20
|