Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..................X..............X.........X.......X........] duration:
1.660864649s trace:
https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed
FAIL (2025-10-31 21:33:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.660888355s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...................X.........................XX.....XX......] duration:
2.389876065s trace:
https://jaeger.dev.storj.io/trace/47742a9fc7eef47b
FAIL (2025-10-31 21:33:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment
Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments...
Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection
Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection
Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection
Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection
Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments...
Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil>
Duration: 2.389905821s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............................................X.....X.X.....] duration:
2.669130438s trace:
https://jaeger.dev.storj.io/trace/4240f607f15f8845
FAIL (2025-10-31 21:33:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded
======LOGS======
load=29731]: Done starting upload
Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments...
Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false
Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment...
Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment.
Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment
Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments...
Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection
Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection
Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments...
Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil>
Duration: 2.669185423s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [................X...............X...........X......XX.......] duration:
3.075302719s trace:
https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b
FAIL (2025-10-31 21:33:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.075334244s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
2.872934731s trace:
https://jaeger.dev.storj.io/trace/563abad3e0219f6f
FAIL (2025-10-31 21:32:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.872960976s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [....................X.........................XX.....XX.....] duration:
2.332431168s trace:
https://jaeger.dev.storj.io/trace/47742a9fc7eef47b
FAIL (2025-10-31 21:32:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment
Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments...
Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection
Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection
Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection
Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection
Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments...
Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil>
Duration: 2.332462734s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.................X...............X...........X......XX......] duration:
2.826534754s trace:
https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b
FAIL (2025-10-31 21:32:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.826567092s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...................X..............X.........X.......X.......] duration:
1.619976558s trace:
https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed
FAIL (2025-10-31 21:32:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.620009773s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............................................X.....X.X....] duration:
2.399178119s trace:
https://jaeger.dev.storj.io/trace/4240f607f15f8845
FAIL (2025-10-31 21:32:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded
======LOGS======
load=29731]: Done starting upload
Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments...
Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false
Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment...
Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment.
Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment
Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments...
Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection
Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection
Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments...
Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil>
Duration: 2.399207567s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.....................X.........................XX.....XX....] duration:
2.599887616s trace:
https://jaeger.dev.storj.io/trace/47742a9fc7eef47b
FAIL (2025-10-31 21:32:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment
Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments...
Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection
Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection
Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection
Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection
Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments...
Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil>
Duration: 2.599940805s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [X...................X..............X.........X.......X......] duration:
1.709676381s trace:
https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed
FAIL (2025-10-31 21:31:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.709703939s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
3.005532635s trace:
https://jaeger.dev.storj.io/trace/563abad3e0219f6f
FAIL (2025-10-31 21:31:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.00555695s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..................X...............X...........X......XX.....] duration:
2.660235417s trace:
https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b
FAIL (2025-10-31 21:31:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.660257126s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
2.98715959s trace:
https://jaeger.dev.storj.io/trace/563abad3e0219f6f
FAIL (2025-10-31 21:31:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.987185106s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [X..................X...............X...........X......XX....] duration:
2.646709817s trace:
https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b
FAIL (2025-10-31 21:31:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.646734671s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [......................X.........................XX.....XX...] duration:
2.408532057s trace:
https://jaeger.dev.storj.io/trace/47742a9fc7eef47b
FAIL (2025-10-31 21:31:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment
Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments...
Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection
Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection
Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection
Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection
Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments...
Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil>
Duration: 2.408562664s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................................................X.....X.X...] duration:
2.546353201s trace:
https://jaeger.dev.storj.io/trace/4240f607f15f8845
FAIL (2025-10-31 21:31:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded
======LOGS======
load=29731]: Done starting upload
Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments...
Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false
Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment...
Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment.
Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment
Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments...
Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection
Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection
Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments...
Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil>
Duration: 2.546381159s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
3.015452901s trace:
https://jaeger.dev.storj.io/trace/563abad3e0219f6f
FAIL (2025-10-31 21:30:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.015478474s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [XX...................X..............X.........X.......X.....] duration:
1.921382847s trace:
https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed
FAIL (2025-10-31 21:30:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.921409302s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.......................X.........................XX.....XX..] duration:
9.800338659s trace:
https://jaeger.dev.storj.io/trace/47742a9fc7eef47b
FAIL (2025-10-31 21:30:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment
Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments...
Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection
Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection
Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection
Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection
Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments...
Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil>
Duration: 9.800380217s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................................X.....X.X..] duration:
2.387425954s trace:
https://jaeger.dev.storj.io/trace/4240f607f15f8845
FAIL (2025-10-31 21:30:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded
======LOGS======
load=29731]: Done starting upload
Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments...
Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false
Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment...
Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment.
Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment
Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments...
Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection
Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection
Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments...
Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil>
Duration: 2.387455558s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [.X..................X...............X...........X......XX...] duration:
2.576773424s trace:
https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b
FAIL (2025-10-31 21:30:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.576800945s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
3.500319793s trace:
https://jaeger.dev.storj.io/trace/563abad3e0219f6f
FAIL (2025-10-31 21:29:45 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.500349727s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [..X..................X...............X...........X......XX..] duration:
2.777175594s trace:
https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b
FAIL (2025-10-31 21:29:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.777206568s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [.XX...................X..............X.........X.......X....] duration:
1.848779846s trace:
https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed
FAIL (2025-10-31 21:29:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.848817686s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................................................X.....X.X.] duration:
2.313202891s trace:
https://jaeger.dev.storj.io/trace/4240f607f15f8845
FAIL (2025-10-31 21:29:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded
======LOGS======
load=29731]: Done starting upload
Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments...
Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false
Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment...
Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment.
Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment
Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments...
Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection
Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection
Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments...
Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil>
Duration: 2.31323022s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [........................X.........................XX.....XX.] duration:
9.239350826s trace:
https://jaeger.dev.storj.io/trace/47742a9fc7eef47b
FAIL (2025-10-31 21:29:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment
Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments...
Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection
Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection
Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection
Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection
Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments...
Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil>
Duration: 9.239387219s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.........................X.........................XX.....XX] duration:
47.877581785s trace:
https://jaeger.dev.storj.io/trace/47742a9fc7eef47b
FAIL (2025-10-31 21:29:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece...
Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment
Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments...
Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil>
Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection
Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection
Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection
Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection
Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold
Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments...
Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil>
Duration: 47.87762954s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [..XX...................X..............X.........X.......X...] duration:
2.211794036s trace:
https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed
FAIL (2025-10-31 21:29:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.211832474s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................................................X.....X.X] duration:
30.004395146s trace:
https://jaeger.dev.storj.io/trace/4240f607f15f8845
FAIL (2025-10-31 21:29:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded
======LOGS======
load=29731]: Done starting upload
Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments...
Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false
Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment...
Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment.
Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece...
Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment
Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments...
Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil>
Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces.
Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection
Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection
Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF
Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold
Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments...
Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil>
Duration: 30.004421179s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [...X..................X...............X...........X......XX.] duration:
2.741804605s trace:
https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b
FAIL (2025-10-31 21:28:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.741838306s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [...XX...................X..............X.........X.......X..] duration:
2.279657523s trace:
https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed
FAIL (2025-10-31 21:28:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.279760868s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
9.470373423s trace:
https://jaeger.dev.storj.io/trace/563abad3e0219f6f
FAIL (2025-10-31 21:28:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.470391733s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [....XX...................X..............X.........X.......X.] duration:
2.377294146s trace:
https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed
FAIL (2025-10-31 21:28:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.377309332s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.001685065s trace:
https://jaeger.dev.storj.io/trace/563abad3e0219f6f
FAIL (2025-10-31 21:28:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001704339s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........................X.........................XX.....X] duration:
1m0.111224516s trace:
https://jaeger.dev.storj.io/trace/7b49d29bc58aa45d
FAIL (2025-10-31 21:28:26 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 10)
======LOGS======
address=45.91.116.6:4007 , piece_id=NOJWNVZRIAGFMW5YPPBKQ6J4DGETKRPAU6RPNPHAE5WUQCKJGUWQ, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:26.995485375 (33h46m29.697077833s) [noise=true , address=77.90.31.13:4636 , piece_id=ULNWUBVNVA2QKDVYKXVPVMMAZYEPQTGAP6NNJTFQ24VNBBDRO5ZA, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:26.996055386 (33h46m29.697647843s) [noise=true , address=77.90.31.16:4751 , piece_id=CHRZGK3FRDAEUBSNH7N4CBQRLC4TID3Q2VCWLXXXU5WJSRAGYWWA, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:26.996684789 (33h46m29.698277249s) [noise=true , address=145.79.227.14:4863 , piece_id=L3RQ3PYW5EXRQJTUCA3VWDKSVTTLCZBNELBLQ654AZCYFXR3HNBA, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:26.997218407 (33h46m29.698810858s) [noise=true , address=77.90.31.19:4868 , piece_id=JIKCEFHZFLELYG5VLENHBYGGEMTV6TM5K2MF2JUD6AW4EVGP23FA, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:26.997655522 (33h46m29.699247973s) [noise=true , address=145.79.227.11:4749 , piece_id=W3DJ5R67PIFBEUKU25JCUNROHZB7GXLZTVOBG6G6XN7EMGPOV75A, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:26.997834897 (33h46m29.699427351s) [noise=true , address=45.91.116.12:4247 , piece_id=NVQXMROQT7XRH24XSO7JUP3QTMESTHSSIDK4SK2JL6VWTFQIPLCQ, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:26.998389784 (33h46m29.699982241s) [noise=true , address=77.90.31.12:4622 , piece_id=FOXLRVQWZGMRLKEC7HUCU6KYI7FWQNWMC67Q3Q2PZAJK4FIE7XWQ, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:26.998845403 (33h46m29.700437857s) [noise=true , address=77.90.31.18:4833 , piece_id=U54ETNPMKYMR6FKLC4ZZPUPDLWNUS7RILSALZCRJPWJK7PHDYUFQ, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:26.999333647 (33h46m29.700926104s) [noise=true , address=77.90.31.21:4884 , piece_id=5PWJTUJS2TTAZYIWLRPDWDKTD7NAPSGJFJXZBPFVWSSWVIEO7K2A, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:26.999873728 (33h46m29.701466186s) [noise=true , address=145.79.227.16:4938 , piece_id=OCVLAOUQHPWDG67KATYSLJZ7YPMMPPWKMASHZKU5HVVGXXEPRZZQ, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:27.000409543 (33h46m29.702002021s) [noise=true , address=45.91.116.11:4133 , piece_id=WUQZ4F6KTQK6ESFVC7MTTSE4ZTNKKDZ2RV6FCBO6YQV633QF3Y6Q, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:27.000945646 (33h46m29.702538103s) [noise=true , address=145.79.227.52:4459 , piece_id=2CHFIGEWFXM6LYGU6UGYZJW7US6NEA6PQDI6V4RQPAUF4JYDBLSA, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:27.001976189 (33h46m29.70356864s) [noise=true , address=77.90.31.14:4673 , piece_id=FGILD5WO4V5SA3KJTDI2MTHOABSQ2JNJEZCZYXILAPDDCREHHDDA, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:27.002676241 (33h46m29.7042687s) [noise=true , address=145.79.227.50:4385 , piece_id=77FVXXPZGKUFGEJICQRUNTQF36XDMGLIYSEPDTIRWQU7YTL5JTLQ, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:27.003321184 (33h46m29.704913627s) [noise=true , address=77.90.31.19:4866 , piece_id=JDZIAOKCP3PQ2YCUUOKKJH5K4CDS65WUTXPCVBDSKMMPJEYPB73Q, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:27.009483416 (33h46m29.711075857s) [noise=true , address=145.79.227.34:4925 , piece_id=CJ3NL2REQEKURJX3H3SNHANP7U6I5QVM6O4LPHBSRG3ZIOCS6VFQ, seg_pos={0 0}, upload=29711]: Uploading piece...
Oct 31 21:27:27.615928986 (33h46m30.317521457s) [upload=29711]: Next returned nil segment
Oct 31 21:27:27.615942090 (33h46m30.317534538s) [upload=29711]: Waiting for error group managing segments...
Oct 31 21:27:27.769770970 (33h46m30.471363419s) [noise=true , address=145.79.227.50:4385 , piece_id=77FVXXPZGKUFGEJICQRUNTQF36XDMGLIYSEPDTIRWQU7YTL5JTLQ, seg_pos={0 0}, upload=29711]: Done uploading piece. err: <nil>
Oct 31 21:27:27.780445732 (33h46m30.482038183s) [noise=true , address=145.79.227.11:4749 , piece_id=W3DJ5R67PIFBEUKU25JCUNROHZB7GXLZTVOBG6G6XN7EMGPOV75A, seg_pos={0 0}, upload=29711]: Done uploading piece. err: <nil>
Oct 31 21:27:27.782252146 (33h46m30.483844595s) [noise=true , address=145.79.227.52:4459 , piece_id=2CHFIGEWFXM6LYGU6UGYZJW7US6NEA6PQDI6V4RQPAUF4JYDBLSA, seg_pos={0 0}, upload=29711]: Done uploading piece. err: <nil>
Oct 31 21:27:27.794844987 (33h46m30.496437441s) [noise=true , address=145.79.227.34:4925 , piece_id=CJ3NL2REQEKURJX3H3SNHANP7U6I5QVM6O4LPHBSRG3ZIOCS6VFQ, seg_pos={0 0}, upload=29711]: Done uploading piece. err: <nil>
Oct 31 21:27:27.798614334 (33h46m30.500206786s) [noise=true , address=145.79.227.14:4863 , piece_id=L3RQ3PYW5EXRQJTUCA3VWDKSVTTLCZBNELBLQ654AZCYFXR3HNBA, seg_pos={0 0}, upload=29711]: Done uploading piece. err: <nil>
Oct 31 21:27:29.047562111 (33h46m31.74915456s) [noise=true , address=145.79.227.16:4938 , piece_id=OCVLAOUQHPWDG67KATYSLJZ7YPMMPPWKMASHZKU5HVVGXXEPRZZQ, seg_pos={0 0}, upload=29711]: Done uploading piece. err: <nil>
Oct 31 21:28:15.576039680 (33h47m18.277632138s) [noise=true , address=77.90.31.13:4636 , piece_id=ULNWUBVNVA2QKDVYKXVPVMMAZYEPQTGAP6NNJTFQ24VNBBDRO5ZA, seg_pos={0 0}, upload=29711]: Done uploading piece. err: <nil>
Oct 31 21:28:21.353613682 (33h47m24.055206145s) [noise=true , address=45.91.116.11:4133 , piece_id=WUQZ4F6KTQK6ESFVC7MTTSE4ZTNKKDZ2RV6FCBO6YQV633QF3Y6Q, seg_pos={0 0}, upload=29711]: Done uploading piece. err: <nil>
Oct 31 21:28:24.549451503 (33h47m27.251043958s) [noise=true , address=45.91.116.14:4194 , piece_id=OGHOR4WCQ242SCEQTLHH3M44QH2NUABCAJLSOT5QEBGGXZPWNJRA, seg_pos={0 0}, upload=29711]: Done uploading piece. err: <nil>
Oct 31 21:28:25.471704553 (33h47m28.173297003s) [noise=true , address=45.91.116.12:4247 , piece_id=NVQXMROQT7XRH24XSO7JUP3QTMESTHSSIDK4SK2JL6VWTFQIPLCQ, seg_pos={0 0}, upload=29711]: Done uploading piece. err: <nil>
Oct 31 21:28:26.583398326 (33h47m29.284990784s) [noise=true , address=77.90.31.14:4673 , piece_id=FGILD5WO4V5SA3KJTDI2MTHOABSQ2JNJEZCZYXILAPDDCREHHDDA, seg_pos={0 0}, upload=29711]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:26.584326119 (33h47m29.285918577s) [noise=true , address=77.90.31.12:4622 , piece_id=FOXLRVQWZGMRLKEC7HUCU6KYI7FWQNWMC67Q3Q2PZAJK4FIE7XWQ, seg_pos={0 0}, upload=29711]: Done uploading piece. err: ecclient: upload failed (node:12MQryCwJkVN1JhFm4QqhypX3WoMDR9rdjELj4oYoVbpa6U823Z, address:77.90.31.12:4622): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:26.584470988 (33h47m29.286063447s) [noise=true , address=77.90.31.18:4833 , piece_id=U54ETNPMKYMR6FKLC4ZZPUPDLWNUS7RILSALZCRJPWJK7PHDYUFQ, seg_pos={0 0}, upload=29711]: Done uploading piece. err: ecclient: upload failed (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp, address:77.90.31.18:4833): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:26.584648071 (33h47m29.286240535s) [noise=true , address=77.90.31.21:4884 , piece_id=5PWJTUJS2TTAZYIWLRPDWDKTD7NAPSGJFJXZBPFVWSSWVIEO7K2A, seg_pos={0 0}, upload=29711]: Done uploading piece. err: ecclient: upload failed (node:12aqgkDxFY3xZa9s2vq1DdCFFX2725aDicrxfvoYuXPfe4ozkHV, address:77.90.31.21:4884): protocol: write tcp 139.59.144.22:55462->77.90.31.21:4884: use of closed network connection; write tcp 139.59.144.22:55462->77.90.31.21:4884: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:55462->77.90.31.21:4884: use of closed network connection
Oct 31 21:28:26.584773245 (33h47m29.28636571s) [noise=true , address=77.90.31.16:4751 , piece_id=CHRZGK3FRDAEUBSNH7N4CBQRLC4TID3Q2VCWLXXXU5WJSRAGYWWA, seg_pos={0 0}, upload=29711]: Done uploading piece. err: ecclient: upload failed (node:12Sj7nPQiL1EmwvnVCpRv1HEeDYFnknTXLNZTvhAUmHDcP2Wpuf, address:77.90.31.16:4751): protocol: write tcp 139.59.144.22:41404->77.90.31.16:4751: use of closed network connection; write tcp 139.59.144.22:41404->77.90.31.16:4751: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41404->77.90.31.16:4751: use of closed network connection
Oct 31 21:28:26.584898540 (33h47m29.286491019s) [noise=true , address=45.91.116.6:4007 , piece_id=NOJWNVZRIAGFMW5YPPBKQ6J4DGETKRPAU6RPNPHAE5WUQCKJGUWQ, seg_pos={0 0}, upload=29711]: Done uploading piece. err: ecclient: upload failed (node:16iqnuPtNs9mPecc5vz71XZ3wpsjvD56UQ5sy9cs2o1uWRmNX6, address:45.91.116.6:4007): protocol: write tcp 139.59.144.22:41462->45.91.116.6:4007: use of closed network connection; write tcp 139.59.144.22:41462->45.91.116.6:4007: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41462->45.91.116.6:4007: use of closed network connection
Oct 31 21:28:26.585221745 (33h47m29.286814206s) [noise=true , address=77.90.31.19:4866 , piece_id=JDZIAOKCP3PQ2YCUUOKKJH5K4CDS65WUTXPCVBDSKMMPJEYPB73Q, seg_pos={0 0}, upload=29711]: Done uploading piece. err: ecclient: upload failed (node:12uCbmPHVXpukjQahs2G6oh7VY2zDMooMAbdgSyHN7Biuesmz3A, address:77.90.31.19:4866): protocol: rpc: dial tcp 77.90.31.19:4866: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:28:26.585413163 (33h47m29.287005611s) [noise=true , address=77.90.31.19:4868 , piece_id=JIKCEFHZFLELYG5VLENHBYGGEMTV6TM5K2MF2JUD6AW4EVGP23FA, seg_pos={0 0}, upload=29711]: Done uploading piece. err: ecclient: upload failed (node:17DGGjPv2UkSCAH8ihb4U7CX2K5mYmPjumSSKc26sbcscrjQX1, address:77.90.31.19:4868): protocol: expected piece hash; context deadline exceeded; EOF
Oct 31 21:28:26.585445725 (33h47m29.287038182s) [seg_pos={0 0}, upload=29711]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil>
Oct 31 21:28:26.692688202 (33h47m29.39428066s) [upload=29711]: Done uploading segments...
Oct 31 21:28:26.692729024 (33h47m29.394321479s) [upload=29711]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10)
Duration: 1m0.111246802s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [....X..................X...............X...........X......XX] duration:
30.001509948s trace:
https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b
FAIL (2025-10-31 21:28:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001522634s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....................................................X.....X.] duration:
2.333136288s trace:
https://jaeger.dev.storj.io/trace/275cdc1ede981965
FAIL (2025-10-31 21:27:59 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:27.226061651 (33h46m29.927654108s) [upload=29713]: Starting upload
Oct 31 21:27:27.226246148 (33h46m29.9278386s) [upload=29713]: Done starting upload
Oct 31 21:27:27.226837254 (33h46m29.928429703s) [upload=29713]: Uploading segments...
Oct 31 21:27:27.227111771 (33h46m29.928704229s) [upload=29713]: Got next segment. Inline: false
Oct 31 21:27:27.227169918 (33h46m29.928762376s) [upload=29713]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:27.227178380 (33h46m29.928770839s) [upload=29713]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:27.374055602 (33h46m30.075648057s) [seg_pos={0 0}, upload=29713]: Begin upload segment...
Oct 31 21:27:27.374317506 (33h46m30.075909961s) [seg_pos={0 0}, upload=29713]: Done begin upload segment.
Oct 31 21:27:27.374385904 (33h46m30.075978362s) [noise=true , address=145.79.227.11:4731 , piece_id=YGMEHBU6CT5HOHABP2GKEZMFY52PUBTB4IL6R3MTOFSB66SGCT7Q, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.375154036 (33h46m30.076746484s) [noise=true , address=145.79.227.14:4840 , piece_id=DMUXP25UDJY2AJOSXIH3KLYNBX6HLPUQTGEPPS3FYTDTPMWKPMSQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.375829066 (33h46m30.077421523s) [noise=true , address=145.79.227.9:4365 , piece_id=FMFJABNJ4XCED5D6O5ZMRG5YR34R737BFKYFVQ2E777D6VPRDLDQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.379136127 (33h46m30.080728585s) [noise=true , address=145.79.227.4:4185 , piece_id=MLCKRK7NLGEITOW2WEKZWLE7ERMTY2GWF7IEJJVEKB562HVTWESA, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.382112272 (33h46m30.083704732s) [noise=true , address=145.79.227.21:4474 , piece_id=ECEYQ7P4F64MRK3LZHECR7CCOLQPNVJ3EF7ZCKR56TAXLO37BAAQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.383684472 (33h46m30.085276938s) [noise=true , address=77.90.31.12:4589 , piece_id=RICGTZADWGZF4DGNHQN5UNIDLXW4SF5O7NJCNIWAKU27UM5FZ6JQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.385132419 (33h46m30.086724878s) [noise=true , address=145.79.227.11:4732 , piece_id=SD4JCSISBRFEE6ROR2VMNAV4NDQ6A4LNOLNGH5JRAJQF55CCPZRQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.385323640 (33h46m30.086916095s) [noise=true , address=145.79.227.25:4618 , piece_id=QBWM3WTQFQBF3PETUB5R56QBOSJG4XXXPWXR5HOE6L7VLHX5TXQA, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.387028639 (33h46m30.088621077s) [noise=true , address=145.79.227.31:4797 , piece_id=5NBX6VKKGOC4QCK4R77ULOHJ2I7QFWPYPEDKV7OKSGJ2FPTQFI7Q, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.388160288 (33h46m30.089752748s) [noise=true , address=145.79.227.20:4439 , piece_id=JXMFLLQRKXXOJJAGAC2YQL6FHGEYGAAROTVEU4TBONIUC5FZSIGQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.389063047 (33h46m30.090655501s) [noise=true , address=77.90.31.21:4877 , piece_id=6EVGPRAUFPLVD2JFS66LUDHQX2RMPASS4IKWT7GITPKKNUKXTQ6A, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.391178409 (33h46m30.092770847s) [noise=true , address=145.79.227.13:4802 , piece_id=SLADCIQND7G7CYCIHTXS56NCIYDRVHGGITUS667ICKTYAYDDZSCA, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.394541968 (33h46m30.096134426s) [noise=true , address=77.90.31.10:4520 , piece_id=45N5RDG2SSPW3PYYKIT4GW7UNP5RYT554IJY4OJ542H2FGP3B5HQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.402827998 (33h46m30.104420455s) [noise=true , address=77.90.31.13:4626 , piece_id=MXKWQGXP6LGT6SPPNVZPKUGH3YCI2SK3JJBQDBHH44VDCE2OFRRQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.410191141 (33h46m30.111783599s) [noise=true , address=145.79.227.24:4583 , piece_id=NTCXFTVUVNXZ4KF5CEGEYKHFZ37TCUWN6IGZABDKHB3YZBPLRIUA, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.411147931 (33h46m30.112740392s) [noise=true , address=145.79.227.10:4402 , piece_id=5VYPAJWXJUWZ7IGBO6UJAYEUXWOGJTYLFTO6AAODG7WPWZVFU6GA, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.995226026 (33h46m30.696818471s) [upload=29713]: Next returned nil segment
Oct 31 21:27:27.995262312 (33h46m30.696854772s) [upload=29713]: Waiting for error group managing segments...
Oct 31 21:27:28.138228406 (33h46m30.839820854s) [noise=true , address=145.79.227.14:4840 , piece_id=DMUXP25UDJY2AJOSXIH3KLYNBX6HLPUQTGEPPS3FYTDTPMWKPMSQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.141180890 (33h46m30.842773349s) [noise=true , address=145.79.227.25:4618 , piece_id=QBWM3WTQFQBF3PETUB5R56QBOSJG4XXXPWXR5HOE6L7VLHX5TXQA, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.143234271 (33h46m30.844826712s) [noise=true , address=145.79.227.13:4802 , piece_id=SLADCIQND7G7CYCIHTXS56NCIYDRVHGGITUS667ICKTYAYDDZSCA, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.146302338 (33h46m30.84789479s) [noise=true , address=145.79.227.31:4797 , piece_id=5NBX6VKKGOC4QCK4R77ULOHJ2I7QFWPYPEDKV7OKSGJ2FPTQFI7Q, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.151005003 (33h46m30.852597452s) [noise=true , address=145.79.227.11:4731 , piece_id=YGMEHBU6CT5HOHABP2GKEZMFY52PUBTB4IL6R3MTOFSB66SGCT7Q, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.155803207 (33h46m30.857395662s) [noise=true , address=145.79.227.11:4732 , piece_id=SD4JCSISBRFEE6ROR2VMNAV4NDQ6A4LNOLNGH5JRAJQF55CCPZRQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.157189527 (33h46m30.858781979s) [noise=true , address=145.79.227.21:4474 , piece_id=ECEYQ7P4F64MRK3LZHECR7CCOLQPNVJ3EF7ZCKR56TAXLO37BAAQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.161288382 (33h46m30.86288082s) [noise=true , address=145.79.227.20:4439 , piece_id=JXMFLLQRKXXOJJAGAC2YQL6FHGEYGAAROTVEU4TBONIUC5FZSIGQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.162709045 (33h46m30.864301483s) [noise=true , address=145.79.227.10:4402 , piece_id=5VYPAJWXJUWZ7IGBO6UJAYEUXWOGJTYLFTO6AAODG7WPWZVFU6GA, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.163313327 (33h46m30.864905763s) [noise=true , address=145.79.227.24:4583 , piece_id=NTCXFTVUVNXZ4KF5CEGEYKHFZ37TCUWN6IGZABDKHB3YZBPLRIUA, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.171648122 (33h46m30.873240579s) [noise=true , address=145.79.227.4:4185 , piece_id=MLCKRK7NLGEITOW2WEKZWLE7ERMTY2GWF7IEJJVEKB562HVTWESA, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.179980885 (33h46m30.881573335s) [noise=true , address=145.79.227.9:4365 , piece_id=FMFJABNJ4XCED5D6O5ZMRG5YR34R737BFKYFVQ2E777D6VPRDLDQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:56.866792752 (33h46m59.56838521s) [noise=true , address=77.90.31.21:4877 , piece_id=6EVGPRAUFPLVD2JFS66LUDHQX2RMPASS4IKWT7GITPKKNUKXTQ6A, seg_pos={0 0}, upload=29713]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:33130->77.90.31.21:4877: use of closed network connection; write tcp 139.59.144.22:33130->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33130->77.90.31.21:4877: use of closed network connection
Oct 31 21:27:56.866939552 (33h46m59.568532019s) [noise=true , address=77.90.31.10:4520 , piece_id=45N5RDG2SSPW3PYYKIT4GW7UNP5RYT554IJY4OJ542H2FGP3B5HQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:57326->77.90.31.10:4520: use of closed network connection; write tcp 139.59.144.22:57326->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57326->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:56.867084068 (33h46m59.568676525s) [noise=true , address=77.90.31.13:4626 , piece_id=MXKWQGXP6LGT6SPPNVZPKUGH3YCI2SK3JJBQDBHH44VDCE2OFRRQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 139.59.144.22:55360->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:56.867169215 (33h46m59.568761678s) [noise=true , address=77.90.31.12:4589 , piece_id=RICGTZADWGZF4DGNHQN5UNIDLXW4SF5O7NJCNIWAKU27UM5FZ6JQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:42594->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:56.867206779 (33h46m59.568799256s) [seg_pos={0 0}, upload=29713]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:56.974711205 (33h46m59.676303653s) [upload=29713]: Done uploading segments...
Oct 31 21:27:56.974756219 (33h46m59.676348667s) [upload=29713]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.333155053s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................................X.....X] duration:
30.109825723s trace:
https://jaeger.dev.storj.io/trace/275cdc1ede981965
FAIL (2025-10-31 21:27:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:27:27.226061651 (33h46m29.927654108s) [upload=29713]: Starting upload
Oct 31 21:27:27.226246148 (33h46m29.9278386s) [upload=29713]: Done starting upload
Oct 31 21:27:27.226837254 (33h46m29.928429703s) [upload=29713]: Uploading segments...
Oct 31 21:27:27.227111771 (33h46m29.928704229s) [upload=29713]: Got next segment. Inline: false
Oct 31 21:27:27.227169918 (33h46m29.928762376s) [upload=29713]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:27:27.227178380 (33h46m29.928770839s) [upload=29713]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:27:27.374055602 (33h46m30.075648057s) [seg_pos={0 0}, upload=29713]: Begin upload segment...
Oct 31 21:27:27.374317506 (33h46m30.075909961s) [seg_pos={0 0}, upload=29713]: Done begin upload segment.
Oct 31 21:27:27.374385904 (33h46m30.075978362s) [noise=true , address=145.79.227.11:4731 , piece_id=YGMEHBU6CT5HOHABP2GKEZMFY52PUBTB4IL6R3MTOFSB66SGCT7Q, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.375154036 (33h46m30.076746484s) [noise=true , address=145.79.227.14:4840 , piece_id=DMUXP25UDJY2AJOSXIH3KLYNBX6HLPUQTGEPPS3FYTDTPMWKPMSQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.375829066 (33h46m30.077421523s) [noise=true , address=145.79.227.9:4365 , piece_id=FMFJABNJ4XCED5D6O5ZMRG5YR34R737BFKYFVQ2E777D6VPRDLDQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.379136127 (33h46m30.080728585s) [noise=true , address=145.79.227.4:4185 , piece_id=MLCKRK7NLGEITOW2WEKZWLE7ERMTY2GWF7IEJJVEKB562HVTWESA, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.382112272 (33h46m30.083704732s) [noise=true , address=145.79.227.21:4474 , piece_id=ECEYQ7P4F64MRK3LZHECR7CCOLQPNVJ3EF7ZCKR56TAXLO37BAAQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.383684472 (33h46m30.085276938s) [noise=true , address=77.90.31.12:4589 , piece_id=RICGTZADWGZF4DGNHQN5UNIDLXW4SF5O7NJCNIWAKU27UM5FZ6JQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.385132419 (33h46m30.086724878s) [noise=true , address=145.79.227.11:4732 , piece_id=SD4JCSISBRFEE6ROR2VMNAV4NDQ6A4LNOLNGH5JRAJQF55CCPZRQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.385323640 (33h46m30.086916095s) [noise=true , address=145.79.227.25:4618 , piece_id=QBWM3WTQFQBF3PETUB5R56QBOSJG4XXXPWXR5HOE6L7VLHX5TXQA, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.387028639 (33h46m30.088621077s) [noise=true , address=145.79.227.31:4797 , piece_id=5NBX6VKKGOC4QCK4R77ULOHJ2I7QFWPYPEDKV7OKSGJ2FPTQFI7Q, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.388160288 (33h46m30.089752748s) [noise=true , address=145.79.227.20:4439 , piece_id=JXMFLLQRKXXOJJAGAC2YQL6FHGEYGAAROTVEU4TBONIUC5FZSIGQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.389063047 (33h46m30.090655501s) [noise=true , address=77.90.31.21:4877 , piece_id=6EVGPRAUFPLVD2JFS66LUDHQX2RMPASS4IKWT7GITPKKNUKXTQ6A, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.391178409 (33h46m30.092770847s) [noise=true , address=145.79.227.13:4802 , piece_id=SLADCIQND7G7CYCIHTXS56NCIYDRVHGGITUS667ICKTYAYDDZSCA, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.394541968 (33h46m30.096134426s) [noise=true , address=77.90.31.10:4520 , piece_id=45N5RDG2SSPW3PYYKIT4GW7UNP5RYT554IJY4OJ542H2FGP3B5HQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.402827998 (33h46m30.104420455s) [noise=true , address=77.90.31.13:4626 , piece_id=MXKWQGXP6LGT6SPPNVZPKUGH3YCI2SK3JJBQDBHH44VDCE2OFRRQ, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.410191141 (33h46m30.111783599s) [noise=true , address=145.79.227.24:4583 , piece_id=NTCXFTVUVNXZ4KF5CEGEYKHFZ37TCUWN6IGZABDKHB3YZBPLRIUA, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.411147931 (33h46m30.112740392s) [noise=true , address=145.79.227.10:4402 , piece_id=5VYPAJWXJUWZ7IGBO6UJAYEUXWOGJTYLFTO6AAODG7WPWZVFU6GA, seg_pos={0 0}, upload=29713]: Uploading piece...
Oct 31 21:27:27.995226026 (33h46m30.696818471s) [upload=29713]: Next returned nil segment
Oct 31 21:27:27.995262312 (33h46m30.696854772s) [upload=29713]: Waiting for error group managing segments...
Oct 31 21:27:28.138228406 (33h46m30.839820854s) [noise=true , address=145.79.227.14:4840 , piece_id=DMUXP25UDJY2AJOSXIH3KLYNBX6HLPUQTGEPPS3FYTDTPMWKPMSQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.141180890 (33h46m30.842773349s) [noise=true , address=145.79.227.25:4618 , piece_id=QBWM3WTQFQBF3PETUB5R56QBOSJG4XXXPWXR5HOE6L7VLHX5TXQA, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.143234271 (33h46m30.844826712s) [noise=true , address=145.79.227.13:4802 , piece_id=SLADCIQND7G7CYCIHTXS56NCIYDRVHGGITUS667ICKTYAYDDZSCA, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.146302338 (33h46m30.84789479s) [noise=true , address=145.79.227.31:4797 , piece_id=5NBX6VKKGOC4QCK4R77ULOHJ2I7QFWPYPEDKV7OKSGJ2FPTQFI7Q, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.151005003 (33h46m30.852597452s) [noise=true , address=145.79.227.11:4731 , piece_id=YGMEHBU6CT5HOHABP2GKEZMFY52PUBTB4IL6R3MTOFSB66SGCT7Q, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.155803207 (33h46m30.857395662s) [noise=true , address=145.79.227.11:4732 , piece_id=SD4JCSISBRFEE6ROR2VMNAV4NDQ6A4LNOLNGH5JRAJQF55CCPZRQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.157189527 (33h46m30.858781979s) [noise=true , address=145.79.227.21:4474 , piece_id=ECEYQ7P4F64MRK3LZHECR7CCOLQPNVJ3EF7ZCKR56TAXLO37BAAQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.161288382 (33h46m30.86288082s) [noise=true , address=145.79.227.20:4439 , piece_id=JXMFLLQRKXXOJJAGAC2YQL6FHGEYGAAROTVEU4TBONIUC5FZSIGQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.162709045 (33h46m30.864301483s) [noise=true , address=145.79.227.10:4402 , piece_id=5VYPAJWXJUWZ7IGBO6UJAYEUXWOGJTYLFTO6AAODG7WPWZVFU6GA, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.163313327 (33h46m30.864905763s) [noise=true , address=145.79.227.24:4583 , piece_id=NTCXFTVUVNXZ4KF5CEGEYKHFZ37TCUWN6IGZABDKHB3YZBPLRIUA, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.171648122 (33h46m30.873240579s) [noise=true , address=145.79.227.4:4185 , piece_id=MLCKRK7NLGEITOW2WEKZWLE7ERMTY2GWF7IEJJVEKB562HVTWESA, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:28.179980885 (33h46m30.881573335s) [noise=true , address=145.79.227.9:4365 , piece_id=FMFJABNJ4XCED5D6O5ZMRG5YR34R737BFKYFVQ2E777D6VPRDLDQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: <nil>
Oct 31 21:27:56.866792752 (33h46m59.56838521s) [noise=true , address=77.90.31.21:4877 , piece_id=6EVGPRAUFPLVD2JFS66LUDHQX2RMPASS4IKWT7GITPKKNUKXTQ6A, seg_pos={0 0}, upload=29713]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 139.59.144.22:33130->77.90.31.21:4877: use of closed network connection; write tcp 139.59.144.22:33130->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:33130->77.90.31.21:4877: use of closed network connection
Oct 31 21:27:56.866939552 (33h46m59.568532019s) [noise=true , address=77.90.31.10:4520 , piece_id=45N5RDG2SSPW3PYYKIT4GW7UNP5RYT554IJY4OJ542H2FGP3B5HQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 139.59.144.22:57326->77.90.31.10:4520: use of closed network connection; write tcp 139.59.144.22:57326->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57326->77.90.31.10:4520: use of closed network connection
Oct 31 21:27:56.867084068 (33h46m59.568676525s) [noise=true , address=77.90.31.13:4626 , piece_id=MXKWQGXP6LGT6SPPNVZPKUGH3YCI2SK3JJBQDBHH44VDCE2OFRRQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: read tcp 139.59.144.22:55360->77.90.31.13:4626: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:56.867169215 (33h46m59.568761678s) [noise=true , address=77.90.31.12:4589 , piece_id=RICGTZADWGZF4DGNHQN5UNIDLXW4SF5O7NJCNIWAKU27UM5FZ6JQ, seg_pos={0 0}, upload=29713]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 139.59.144.22:42594->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:27:56.867206779 (33h46m59.568799256s) [seg_pos={0 0}, upload=29713]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:27:56.974711205 (33h46m59.676303653s) [upload=29713]: Done uploading segments...
Oct 31 21:27:56.974756219 (33h46m59.676348667s) [upload=29713]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.109846469s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
6 out of 60 is failed (threshold=3) [X....X..................X...............X...........X......X] duration:
30.001234466s trace:
https://jaeger.dev.storj.io/trace/3b795fc620b573e2
FAIL (2025-10-31 21:27:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001245751s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [X....XX...................X..............X.........X.......X] duration:
5.001503075s trace:
https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed
FAIL (2025-10-31 21:27:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001526859s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
2.451814719s trace:
https://jaeger.dev.storj.io/trace/62df97e1df2d7a
FAIL (2025-10-31 21:26:52 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:22:07.085611190 (33h41m9.787203641s) [upload=29636]: Starting upload
Oct 31 21:22:07.085688329 (33h41m9.787280767s) [upload=29636]: Done starting upload
Oct 31 21:22:07.085774737 (33h41m9.78736719s) [upload=29636]: Uploading segments...
Oct 31 21:22:07.086049239 (33h41m9.787641687s) [upload=29636]: Got next segment. Inline: false
Oct 31 21:22:07.086139291 (33h41m9.78773174s) [upload=29636]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:07.086145090 (33h41m9.787737538s) [upload=29636]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:07.198887636 (33h41m9.900480077s) [seg_pos={0 0}, upload=29636]: Begin upload segment...
Oct 31 21:22:07.198943347 (33h41m9.900535799s) [seg_pos={0 0}, upload=29636]: Done begin upload segment.
Oct 31 21:22:07.198976223 (33h41m9.900568674s) [noise=true , address=145.79.227.27:4689 , piece_id=E2674435DTU7SSPTAFGMRLBKZNPZAOIFMBDN5N734CP4RG23SVCA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199094043 (33h41m9.900686499s) [noise=true , address=145.79.227.15:4875 , piece_id=GXNV3O7D7URXRTLWVXMFMVTXGHZ4PAZMGQLI3L5SNUX3LDV7OXJA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199764319 (33h41m9.901356931s) [noise=true , address=145.79.227.14:4840 , piece_id=MKR2JQOAFU6BA6JUG4LTAICYEDRI367JLXCNW6EPZWIJR2ZNKXYA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199979140 (33h41m9.901571602s) [noise=true , address=145.79.227.28:4725 , piece_id=2UEZO2DIIDOVE4V6TXLKZAWXTOUNTIJKCBOP6JHREEEYBUWEXAUA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.201094736 (33h41m9.902687196s) [noise=true , address=145.79.227.10:4401 , piece_id=YCZ6LL4ECE2QKCLEAF77L3C72VWCZ6IQVJLJ77RIACX4LIK2IJJA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.201150898 (33h41m9.902743361s) [noise=true , address=145.79.227.32:4834 , piece_id=LBXTDM2QLBWFNDJ5272FY546TCRWC3IFYRNOBCCIS3KQDFMVJDFQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.202300922 (33h41m9.903893383s) [noise=true , address=45.91.116.4:4940 , piece_id=7TW5URZLYHRH7MQX3PZGMYPKJS3SYTXCJJDSKCOQZHB2HIS6J4OA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.202347708 (33h41m9.903940165s) [noise=true , address=77.90.31.12:4590 , piece_id=WQQC3535ERLWQOD6WCWJF3DUY4FY4CZF7KPD6Y2K3Q2NYQCP45JA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.203610241 (33h41m9.905202697s) [noise=true , address=145.79.227.31:4798 , piece_id=QIYHHYEZGJEMENOVRJIDH7722MX2HI7XAGWUSFKIV3YNHKVPVD2Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.204358358 (33h41m9.905950816s) [noise=true , address=145.79.227.9:4367 , piece_id=Q2S6IK6OE4535GKC6J55MV3N45EUDWGOYKWS3ISQSZJKW2WJCD6Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.204958129 (33h41m9.906550588s) [noise=true , address=45.91.116.4:4939 , piece_id=VYJO7TDD6SANMOZ7BB62KYLU72ACLUIZI7IHFV6I2PDAV7K7WLCQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.206076686 (33h41m9.907669138s) [noise=true , address=145.79.227.30:4763 , piece_id=ET72Q2LHTA3HQY2GQNCO2ZEOBP526EOWM72TL5XRQYOLLARFPZLQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.207450031 (33h41m9.909042495s) [noise=true , address=45.91.116.8:4045 , piece_id=B6C3QMJBWPBBJBTSYFP67R5WOVVGIGOYN6B6P5BAW35FXYEVPW3A, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.208460871 (33h41m9.910053312s) [noise=true , address=145.79.227.15:4873 , piece_id=FIIZHJVUTZ42TATHLHZEWSRMVUFB7DYTWGIABNR4LJQYVH4SKQ5A, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.209461830 (33h41m9.911054339s) [noise=true , address=145.79.227.11:4729 , piece_id=ZYPWL4SEIZHXYOAW4I5JOCPXDTHRCMO2FXYGA4ZUNHA2CSZ6DG5Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.210046720 (33h41m9.911639172s) [noise=true , address=145.79.227.30:4762 , piece_id=4BL5PJ3S2OC7PVSILA4IQCFT4JW6W5GBOXI7WRAS52DNBCF3DIHQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.808217579 (33h41m10.509810059s) [upload=29636]: Next returned nil segment
Oct 31 21:22:07.808236009 (33h41m10.509828464s) [upload=29636]: Waiting for error group managing segments...
Oct 31 21:22:07.951052914 (33h41m10.65264537s) [noise=true , address=145.79.227.32:4834 , piece_id=LBXTDM2QLBWFNDJ5272FY546TCRWC3IFYRNOBCCIS3KQDFMVJDFQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.951804679 (33h41m10.653397134s) [noise=true , address=145.79.227.15:4875 , piece_id=GXNV3O7D7URXRTLWVXMFMVTXGHZ4PAZMGQLI3L5SNUX3LDV7OXJA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.951868872 (33h41m10.653461324s) [noise=true , address=145.79.227.31:4798 , piece_id=QIYHHYEZGJEMENOVRJIDH7722MX2HI7XAGWUSFKIV3YNHKVPVD2Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952470935 (33h41m10.654063385s) [noise=true , address=145.79.227.30:4762 , piece_id=4BL5PJ3S2OC7PVSILA4IQCFT4JW6W5GBOXI7WRAS52DNBCF3DIHQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952500937 (33h41m10.654093387s) [noise=true , address=145.79.227.27:4689 , piece_id=E2674435DTU7SSPTAFGMRLBKZNPZAOIFMBDN5N734CP4RG23SVCA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952980719 (33h41m10.654573168s) [noise=true , address=145.79.227.28:4725 , piece_id=2UEZO2DIIDOVE4V6TXLKZAWXTOUNTIJKCBOP6JHREEEYBUWEXAUA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.953254355 (33h41m10.654846814s) [noise=true , address=145.79.227.11:4729 , piece_id=ZYPWL4SEIZHXYOAW4I5JOCPXDTHRCMO2FXYGA4ZUNHA2CSZ6DG5Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.953604050 (33h41m10.6551965s) [noise=true , address=145.79.227.15:4873 , piece_id=FIIZHJVUTZ42TATHLHZEWSRMVUFB7DYTWGIABNR4LJQYVH4SKQ5A, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.954075643 (33h41m10.6556681s) [noise=true , address=145.79.227.9:4367 , piece_id=Q2S6IK6OE4535GKC6J55MV3N45EUDWGOYKWS3ISQSZJKW2WJCD6Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.958770427 (33h41m10.660362881s) [noise=true , address=145.79.227.10:4401 , piece_id=YCZ6LL4ECE2QKCLEAF77L3C72VWCZ6IQVJLJ77RIACX4LIK2IJJA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.965022314 (33h41m10.666614767s) [noise=true , address=145.79.227.30:4763 , piece_id=ET72Q2LHTA3HQY2GQNCO2ZEOBP526EOWM72TL5XRQYOLLARFPZLQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.976650683 (33h41m10.678243138s) [noise=true , address=145.79.227.14:4840 , piece_id=MKR2JQOAFU6BA6JUG4LTAICYEDRI367JLXCNW6EPZWIJR2ZNKXYA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:36.799544661 (33h41m39.501137117s) [noise=true , address=45.91.116.4:4939 , piece_id=VYJO7TDD6SANMOZ7BB62KYLU72ACLUIZI7IHFV6I2PDAV7K7WLCQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection
Oct 31 21:22:36.799605331 (33h41m39.50119779s) [noise=true , address=77.90.31.12:4590 , piece_id=WQQC3535ERLWQOD6WCWJF3DUY4FY4CZF7KPD6Y2K3Q2NYQCP45JA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection; write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection
Oct 31 21:22:36.799747990 (33h41m39.501340451s) [noise=true , address=45.91.116.4:4940 , piece_id=7TW5URZLYHRH7MQX3PZGMYPKJS3SYTXCJJDSKCOQZHB2HIS6J4OA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 139.59.144.22:36232->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:36.799846589 (33h41m39.501439048s) [noise=true , address=45.91.116.8:4045 , piece_id=B6C3QMJBWPBBJBTSYFP67R5WOVVGIGOYN6B6P5BAW35FXYEVPW3A, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:39300->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:36.799901279 (33h41m39.501493715s) [seg_pos={0 0}, upload=29636]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:22:36.906137907 (33h41m39.607730362s) [upload=29636]: Done uploading segments...
Oct 31 21:22:36.906171689 (33h41m39.60776414s) [upload=29636]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.451843327s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X....X..................X...............X...........X......] duration:
2.57748405s trace:
https://jaeger.dev.storj.io/trace/5f0044a50b568df8
FAIL (2025-10-31 21:26:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.57751773s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................X.........................XX.....] duration:
2.318718193s trace:
https://jaeger.dev.storj.io/trace/3173f13c3ae66d35
FAIL (2025-10-31 21:26:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
NOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.439072473 (33h41m54.140664943s) [noise=true , address=145.79.227.33:4888 , piece_id=55HIHYQGP4PYV47T3EOPUOOWH74Q2AC66ZJAVNMNY2NUDV5DM7QA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.440028262 (33h41m54.141620722s) [noise=true , address=145.79.227.36:4549 , piece_id=OPMGCVNCWD44POIDEO46A4ZKUJ4BQE7UDPSIF3UAIDWZP2QJRC4A, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.440879298 (33h41m54.142471755s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.441210726 (33h41m54.142803185s) [noise=true , address=77.90.31.19:4865 , piece_id=ZDUGAOKKTEBBUF4S7ZFOGSBITGLV5BUW2RHCVHVSDVPEYNTEWDOQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.441758919 (33h41m54.143351377s) [noise=true , address=77.90.31.12:4609 , piece_id=N4LUJ74AFB7JAE4ZL66HLKG746LGHKNFBY7TZTSTQOTIBPQN7I2Q, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442265638 (33h41m54.143858095s) [noise=true , address=45.91.116.10:4113 , piece_id=PIGRIKVRZCWODXJVZ4SIORYLZYWRMMB2EXQRZGJZRK3JCXLWRKOA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442303271 (33h41m54.14389573s) [noise=true , address=77.90.31.21:4910 , piece_id=K4ERYFVADFU242IHSKR6BGQJHLUNSQPIHNFHPA7QLNF7JSJSACZQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442656746 (33h41m54.144249202s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442782370 (33h41m54.144374829s) [noise=true , address=45.91.116.9:4286 , piece_id=3T3JM6QB6GN4OUSP2LN7ICHBZL3CO5GW56ODFBB7KVFNXSHRD52A, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443021968 (33h41m54.144614421s) [noise=true , address=45.91.116.8:4057 , piece_id=BVGJB2U2MHIEUOUOPF5GF6SY63DVJQ54LLI4W7SND3DZX56MZZUQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443252911 (33h41m54.144845368s) [noise=true , address=77.90.31.21:4911 , piece_id=FMIATBPBQP237AJC3IMMG7KC74M424BQ2KYSXJQOZHUMIIXXAVWQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443438074 (33h41m54.145030531s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443753610 (33h41m54.145346067s) [noise=true , address=145.79.227.6:4269 , piece_id=ZEEU26BQ4H4WTK4TMHGJYQPTQMQNQOWJGAECEINQCQDKTKATZ7DA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443851896 (33h41m54.145444348s) [noise=true , address=45.91.116.5:4308 , piece_id=FTWXK442NYHAARIK7REHW7GHHQ6K4URWJD75K6QNYXCI3S3BMKWA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.444315561 (33h41m54.145908018s) [noise=true , address=45.91.116.12:4242 , piece_id=TXGM7MH6RQ33AHDFWD4X3DSNZGL4DP4WBEFUFQXW5HEUPE56JWYA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.445026431 (33h41m54.14661889s) [noise=true , address=45.91.116.13:4177 , piece_id=XREU7IAJVWS2MJIEEEFOVKQOIAXJ3DGSX3NDA7KJUL6IUIXMX4PQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.445715106 (33h41m54.147307563s) [noise=true , address=145.79.227.52:4461 , piece_id=LQJVVKNJNZOVEXRW7Y5KGLX4IDFZMCG65UN3BAVPJEIHWN3NIWQA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:52.002260082 (33h41m54.703852532s) [upload=29647]: Next returned nil segment
Oct 31 21:22:52.002281375 (33h41m54.703873836s) [upload=29647]: Waiting for error group managing segments...
Oct 31 21:22:52.166910441 (33h41m54.868502889s) [noise=true , address=145.79.227.33:4888 , piece_id=55HIHYQGP4PYV47T3EOPUOOWH74Q2AC66ZJAVNMNY2NUDV5DM7QA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.171085961 (33h41m54.872678409s) [noise=true , address=145.79.227.36:4549 , piece_id=OPMGCVNCWD44POIDEO46A4ZKUJ4BQE7UDPSIF3UAIDWZP2QJRC4A, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.173190935 (33h41m54.874783383s) [noise=true , address=145.79.227.52:4461 , piece_id=LQJVVKNJNZOVEXRW7Y5KGLX4IDFZMCG65UN3BAVPJEIHWN3NIWQA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.174554840 (33h41m54.876147316s) [noise=true , address=145.79.227.6:4269 , piece_id=ZEEU26BQ4H4WTK4TMHGJYQPTQMQNQOWJGAECEINQCQDKTKATZ7DA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.870199575 (33h41m55.571792024s) [noise=true , address=45.91.116.8:4057 , piece_id=BVGJB2U2MHIEUOUOPF5GF6SY63DVJQ54LLI4W7SND3DZX56MZZUQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.932896280 (33h41m55.634488728s) [noise=true , address=45.91.116.5:4308 , piece_id=FTWXK442NYHAARIK7REHW7GHHQ6K4URWJD75K6QNYXCI3S3BMKWA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.996436348 (33h41m55.698028803s) [noise=true , address=45.91.116.13:4177 , piece_id=XREU7IAJVWS2MJIEEEFOVKQOIAXJ3DGSX3NDA7KJUL6IUIXMX4PQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:53.303664204 (33h41m56.005256643s) [noise=true , address=45.91.116.10:4113 , piece_id=PIGRIKVRZCWODXJVZ4SIORYLZYWRMMB2EXQRZGJZRK3JCXLWRKOA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:53.934904546 (33h41m56.636497004s) [noise=true , address=77.90.31.21:4910 , piece_id=K4ERYFVADFU242IHSKR6BGQJHLUNSQPIHNFHPA7QLNF7JSJSACZQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:54.224811350 (33h41m56.926403798s) [noise=true , address=77.90.31.12:4609 , piece_id=N4LUJ74AFB7JAE4ZL66HLKG746LGHKNFBY7TZTSTQOTIBPQN7I2Q, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:54.652552655 (33h41m57.35414511s) [noise=true , address=45.91.116.9:4286 , piece_id=3T3JM6QB6GN4OUSP2LN7ICHBZL3CO5GW56ODFBB7KVFNXSHRD52A, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:55.813839187 (33h41m58.515431634s) [noise=true , address=45.91.116.12:4242 , piece_id=TXGM7MH6RQ33AHDFWD4X3DSNZGL4DP4WBEFUFQXW5HEUPE56JWYA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:56.776728453 (33h41m59.4783209s) [noise=true , address=77.90.31.21:4911 , piece_id=FMIATBPBQP237AJC3IMMG7KC74M424BQ2KYSXJQOZHUMIIXXAVWQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:58.637065006 (33h42m1.338657444s) [noise=true , address=77.90.31.19:4865 , piece_id=ZDUGAOKKTEBBUF4S7ZFOGSBITGLV5BUW2RHCVHVSDVPEYNTEWDOQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:58.637090306 (33h42m1.338682758s) [seg_pos={0 0}, upload=29647]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:22:58.637316080 (33h42m1.338908539s) [noise=true , address=77.90.31.17:4787 , piece_id=V3EHNOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637326708 (33h42m1.338919172s) [noise=true , address=77.90.31.17:4787 , piece_id=V3EHNOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637380935 (33h42m1.338973393s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637394866 (33h42m1.338987321s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637446506 (33h42m1.339038963s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637456777 (33h42m1.339049231s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637522129 (33h42m1.339114607s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637528827 (33h42m1.339121275s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637546736 (33h42m1.339139173s) [seg_pos={0 0}, upload=29647]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:22:58.637649415 (33h42m1.339241873s) [upload=29647]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:22:58.637657672 (33h42m1.339250145s) [upload=29647]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:22:58.787848722 (33h42m1.489441173s) [upload=29647]: Done uploading segments...
Oct 31 21:22:58.787868799 (33h42m1.489461251s) [upload=29647]: Upload finished. err: <nil>
Duration: 2.318747656s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [.X....XX...................X..............X.........X.......] duration:
2.137521394s trace:
https://jaeger.dev.storj.io/trace/5b5403da1e956ee2
FAIL (2025-10-31 21:26:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.137855438s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [............................X.........................XX....] duration:
2.34385571s trace:
https://jaeger.dev.storj.io/trace/3173f13c3ae66d35
FAIL (2025-10-31 21:26:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
NOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.439072473 (33h41m54.140664943s) [noise=true , address=145.79.227.33:4888 , piece_id=55HIHYQGP4PYV47T3EOPUOOWH74Q2AC66ZJAVNMNY2NUDV5DM7QA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.440028262 (33h41m54.141620722s) [noise=true , address=145.79.227.36:4549 , piece_id=OPMGCVNCWD44POIDEO46A4ZKUJ4BQE7UDPSIF3UAIDWZP2QJRC4A, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.440879298 (33h41m54.142471755s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.441210726 (33h41m54.142803185s) [noise=true , address=77.90.31.19:4865 , piece_id=ZDUGAOKKTEBBUF4S7ZFOGSBITGLV5BUW2RHCVHVSDVPEYNTEWDOQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.441758919 (33h41m54.143351377s) [noise=true , address=77.90.31.12:4609 , piece_id=N4LUJ74AFB7JAE4ZL66HLKG746LGHKNFBY7TZTSTQOTIBPQN7I2Q, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442265638 (33h41m54.143858095s) [noise=true , address=45.91.116.10:4113 , piece_id=PIGRIKVRZCWODXJVZ4SIORYLZYWRMMB2EXQRZGJZRK3JCXLWRKOA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442303271 (33h41m54.14389573s) [noise=true , address=77.90.31.21:4910 , piece_id=K4ERYFVADFU242IHSKR6BGQJHLUNSQPIHNFHPA7QLNF7JSJSACZQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442656746 (33h41m54.144249202s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442782370 (33h41m54.144374829s) [noise=true , address=45.91.116.9:4286 , piece_id=3T3JM6QB6GN4OUSP2LN7ICHBZL3CO5GW56ODFBB7KVFNXSHRD52A, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443021968 (33h41m54.144614421s) [noise=true , address=45.91.116.8:4057 , piece_id=BVGJB2U2MHIEUOUOPF5GF6SY63DVJQ54LLI4W7SND3DZX56MZZUQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443252911 (33h41m54.144845368s) [noise=true , address=77.90.31.21:4911 , piece_id=FMIATBPBQP237AJC3IMMG7KC74M424BQ2KYSXJQOZHUMIIXXAVWQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443438074 (33h41m54.145030531s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443753610 (33h41m54.145346067s) [noise=true , address=145.79.227.6:4269 , piece_id=ZEEU26BQ4H4WTK4TMHGJYQPTQMQNQOWJGAECEINQCQDKTKATZ7DA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443851896 (33h41m54.145444348s) [noise=true , address=45.91.116.5:4308 , piece_id=FTWXK442NYHAARIK7REHW7GHHQ6K4URWJD75K6QNYXCI3S3BMKWA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.444315561 (33h41m54.145908018s) [noise=true , address=45.91.116.12:4242 , piece_id=TXGM7MH6RQ33AHDFWD4X3DSNZGL4DP4WBEFUFQXW5HEUPE56JWYA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.445026431 (33h41m54.14661889s) [noise=true , address=45.91.116.13:4177 , piece_id=XREU7IAJVWS2MJIEEEFOVKQOIAXJ3DGSX3NDA7KJUL6IUIXMX4PQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.445715106 (33h41m54.147307563s) [noise=true , address=145.79.227.52:4461 , piece_id=LQJVVKNJNZOVEXRW7Y5KGLX4IDFZMCG65UN3BAVPJEIHWN3NIWQA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:52.002260082 (33h41m54.703852532s) [upload=29647]: Next returned nil segment
Oct 31 21:22:52.002281375 (33h41m54.703873836s) [upload=29647]: Waiting for error group managing segments...
Oct 31 21:22:52.166910441 (33h41m54.868502889s) [noise=true , address=145.79.227.33:4888 , piece_id=55HIHYQGP4PYV47T3EOPUOOWH74Q2AC66ZJAVNMNY2NUDV5DM7QA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.171085961 (33h41m54.872678409s) [noise=true , address=145.79.227.36:4549 , piece_id=OPMGCVNCWD44POIDEO46A4ZKUJ4BQE7UDPSIF3UAIDWZP2QJRC4A, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.173190935 (33h41m54.874783383s) [noise=true , address=145.79.227.52:4461 , piece_id=LQJVVKNJNZOVEXRW7Y5KGLX4IDFZMCG65UN3BAVPJEIHWN3NIWQA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.174554840 (33h41m54.876147316s) [noise=true , address=145.79.227.6:4269 , piece_id=ZEEU26BQ4H4WTK4TMHGJYQPTQMQNQOWJGAECEINQCQDKTKATZ7DA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.870199575 (33h41m55.571792024s) [noise=true , address=45.91.116.8:4057 , piece_id=BVGJB2U2MHIEUOUOPF5GF6SY63DVJQ54LLI4W7SND3DZX56MZZUQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.932896280 (33h41m55.634488728s) [noise=true , address=45.91.116.5:4308 , piece_id=FTWXK442NYHAARIK7REHW7GHHQ6K4URWJD75K6QNYXCI3S3BMKWA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.996436348 (33h41m55.698028803s) [noise=true , address=45.91.116.13:4177 , piece_id=XREU7IAJVWS2MJIEEEFOVKQOIAXJ3DGSX3NDA7KJUL6IUIXMX4PQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:53.303664204 (33h41m56.005256643s) [noise=true , address=45.91.116.10:4113 , piece_id=PIGRIKVRZCWODXJVZ4SIORYLZYWRMMB2EXQRZGJZRK3JCXLWRKOA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:53.934904546 (33h41m56.636497004s) [noise=true , address=77.90.31.21:4910 , piece_id=K4ERYFVADFU242IHSKR6BGQJHLUNSQPIHNFHPA7QLNF7JSJSACZQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:54.224811350 (33h41m56.926403798s) [noise=true , address=77.90.31.12:4609 , piece_id=N4LUJ74AFB7JAE4ZL66HLKG746LGHKNFBY7TZTSTQOTIBPQN7I2Q, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:54.652552655 (33h41m57.35414511s) [noise=true , address=45.91.116.9:4286 , piece_id=3T3JM6QB6GN4OUSP2LN7ICHBZL3CO5GW56ODFBB7KVFNXSHRD52A, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:55.813839187 (33h41m58.515431634s) [noise=true , address=45.91.116.12:4242 , piece_id=TXGM7MH6RQ33AHDFWD4X3DSNZGL4DP4WBEFUFQXW5HEUPE56JWYA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:56.776728453 (33h41m59.4783209s) [noise=true , address=77.90.31.21:4911 , piece_id=FMIATBPBQP237AJC3IMMG7KC74M424BQ2KYSXJQOZHUMIIXXAVWQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:58.637065006 (33h42m1.338657444s) [noise=true , address=77.90.31.19:4865 , piece_id=ZDUGAOKKTEBBUF4S7ZFOGSBITGLV5BUW2RHCVHVSDVPEYNTEWDOQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:58.637090306 (33h42m1.338682758s) [seg_pos={0 0}, upload=29647]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:22:58.637316080 (33h42m1.338908539s) [noise=true , address=77.90.31.17:4787 , piece_id=V3EHNOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637326708 (33h42m1.338919172s) [noise=true , address=77.90.31.17:4787 , piece_id=V3EHNOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637380935 (33h42m1.338973393s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637394866 (33h42m1.338987321s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637446506 (33h42m1.339038963s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637456777 (33h42m1.339049231s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637522129 (33h42m1.339114607s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637528827 (33h42m1.339121275s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637546736 (33h42m1.339139173s) [seg_pos={0 0}, upload=29647]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:22:58.637649415 (33h42m1.339241873s) [upload=29647]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:22:58.637657672 (33h42m1.339250145s) [upload=29647]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:22:58.787848722 (33h42m1.489441173s) [upload=29647]: Done uploading segments...
Oct 31 21:22:58.787868799 (33h42m1.489461251s) [upload=29647]: Upload finished. err: <nil>
Duration: 2.344184413s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X....X..................X...............X...........X.....] duration:
2.53557139s trace:
https://jaeger.dev.storj.io/trace/5f0044a50b568df8
FAIL (2025-10-31 21:25:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.535630249s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............................X.........................XX...] duration:
2.436624836s trace:
https://jaeger.dev.storj.io/trace/3173f13c3ae66d35
FAIL (2025-10-31 21:25:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
NOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.439072473 (33h41m54.140664943s) [noise=true , address=145.79.227.33:4888 , piece_id=55HIHYQGP4PYV47T3EOPUOOWH74Q2AC66ZJAVNMNY2NUDV5DM7QA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.440028262 (33h41m54.141620722s) [noise=true , address=145.79.227.36:4549 , piece_id=OPMGCVNCWD44POIDEO46A4ZKUJ4BQE7UDPSIF3UAIDWZP2QJRC4A, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.440879298 (33h41m54.142471755s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.441210726 (33h41m54.142803185s) [noise=true , address=77.90.31.19:4865 , piece_id=ZDUGAOKKTEBBUF4S7ZFOGSBITGLV5BUW2RHCVHVSDVPEYNTEWDOQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.441758919 (33h41m54.143351377s) [noise=true , address=77.90.31.12:4609 , piece_id=N4LUJ74AFB7JAE4ZL66HLKG746LGHKNFBY7TZTSTQOTIBPQN7I2Q, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442265638 (33h41m54.143858095s) [noise=true , address=45.91.116.10:4113 , piece_id=PIGRIKVRZCWODXJVZ4SIORYLZYWRMMB2EXQRZGJZRK3JCXLWRKOA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442303271 (33h41m54.14389573s) [noise=true , address=77.90.31.21:4910 , piece_id=K4ERYFVADFU242IHSKR6BGQJHLUNSQPIHNFHPA7QLNF7JSJSACZQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442656746 (33h41m54.144249202s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.442782370 (33h41m54.144374829s) [noise=true , address=45.91.116.9:4286 , piece_id=3T3JM6QB6GN4OUSP2LN7ICHBZL3CO5GW56ODFBB7KVFNXSHRD52A, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443021968 (33h41m54.144614421s) [noise=true , address=45.91.116.8:4057 , piece_id=BVGJB2U2MHIEUOUOPF5GF6SY63DVJQ54LLI4W7SND3DZX56MZZUQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443252911 (33h41m54.144845368s) [noise=true , address=77.90.31.21:4911 , piece_id=FMIATBPBQP237AJC3IMMG7KC74M424BQ2KYSXJQOZHUMIIXXAVWQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443438074 (33h41m54.145030531s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443753610 (33h41m54.145346067s) [noise=true , address=145.79.227.6:4269 , piece_id=ZEEU26BQ4H4WTK4TMHGJYQPTQMQNQOWJGAECEINQCQDKTKATZ7DA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.443851896 (33h41m54.145444348s) [noise=true , address=45.91.116.5:4308 , piece_id=FTWXK442NYHAARIK7REHW7GHHQ6K4URWJD75K6QNYXCI3S3BMKWA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.444315561 (33h41m54.145908018s) [noise=true , address=45.91.116.12:4242 , piece_id=TXGM7MH6RQ33AHDFWD4X3DSNZGL4DP4WBEFUFQXW5HEUPE56JWYA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.445026431 (33h41m54.14661889s) [noise=true , address=45.91.116.13:4177 , piece_id=XREU7IAJVWS2MJIEEEFOVKQOIAXJ3DGSX3NDA7KJUL6IUIXMX4PQ, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:51.445715106 (33h41m54.147307563s) [noise=true , address=145.79.227.52:4461 , piece_id=LQJVVKNJNZOVEXRW7Y5KGLX4IDFZMCG65UN3BAVPJEIHWN3NIWQA, seg_pos={0 0}, upload=29647]: Uploading piece...
Oct 31 21:22:52.002260082 (33h41m54.703852532s) [upload=29647]: Next returned nil segment
Oct 31 21:22:52.002281375 (33h41m54.703873836s) [upload=29647]: Waiting for error group managing segments...
Oct 31 21:22:52.166910441 (33h41m54.868502889s) [noise=true , address=145.79.227.33:4888 , piece_id=55HIHYQGP4PYV47T3EOPUOOWH74Q2AC66ZJAVNMNY2NUDV5DM7QA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.171085961 (33h41m54.872678409s) [noise=true , address=145.79.227.36:4549 , piece_id=OPMGCVNCWD44POIDEO46A4ZKUJ4BQE7UDPSIF3UAIDWZP2QJRC4A, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.173190935 (33h41m54.874783383s) [noise=true , address=145.79.227.52:4461 , piece_id=LQJVVKNJNZOVEXRW7Y5KGLX4IDFZMCG65UN3BAVPJEIHWN3NIWQA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.174554840 (33h41m54.876147316s) [noise=true , address=145.79.227.6:4269 , piece_id=ZEEU26BQ4H4WTK4TMHGJYQPTQMQNQOWJGAECEINQCQDKTKATZ7DA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.870199575 (33h41m55.571792024s) [noise=true , address=45.91.116.8:4057 , piece_id=BVGJB2U2MHIEUOUOPF5GF6SY63DVJQ54LLI4W7SND3DZX56MZZUQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.932896280 (33h41m55.634488728s) [noise=true , address=45.91.116.5:4308 , piece_id=FTWXK442NYHAARIK7REHW7GHHQ6K4URWJD75K6QNYXCI3S3BMKWA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:52.996436348 (33h41m55.698028803s) [noise=true , address=45.91.116.13:4177 , piece_id=XREU7IAJVWS2MJIEEEFOVKQOIAXJ3DGSX3NDA7KJUL6IUIXMX4PQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:53.303664204 (33h41m56.005256643s) [noise=true , address=45.91.116.10:4113 , piece_id=PIGRIKVRZCWODXJVZ4SIORYLZYWRMMB2EXQRZGJZRK3JCXLWRKOA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:53.934904546 (33h41m56.636497004s) [noise=true , address=77.90.31.21:4910 , piece_id=K4ERYFVADFU242IHSKR6BGQJHLUNSQPIHNFHPA7QLNF7JSJSACZQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:54.224811350 (33h41m56.926403798s) [noise=true , address=77.90.31.12:4609 , piece_id=N4LUJ74AFB7JAE4ZL66HLKG746LGHKNFBY7TZTSTQOTIBPQN7I2Q, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:54.652552655 (33h41m57.35414511s) [noise=true , address=45.91.116.9:4286 , piece_id=3T3JM6QB6GN4OUSP2LN7ICHBZL3CO5GW56ODFBB7KVFNXSHRD52A, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:55.813839187 (33h41m58.515431634s) [noise=true , address=45.91.116.12:4242 , piece_id=TXGM7MH6RQ33AHDFWD4X3DSNZGL4DP4WBEFUFQXW5HEUPE56JWYA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:56.776728453 (33h41m59.4783209s) [noise=true , address=77.90.31.21:4911 , piece_id=FMIATBPBQP237AJC3IMMG7KC74M424BQ2KYSXJQOZHUMIIXXAVWQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:58.637065006 (33h42m1.338657444s) [noise=true , address=77.90.31.19:4865 , piece_id=ZDUGAOKKTEBBUF4S7ZFOGSBITGLV5BUW2RHCVHVSDVPEYNTEWDOQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: <nil>
Oct 31 21:22:58.637090306 (33h42m1.338682758s) [seg_pos={0 0}, upload=29647]: Segment reached optimal threshold of 14 pieces.
Oct 31 21:22:58.637316080 (33h42m1.338908539s) [noise=true , address=77.90.31.17:4787 , piece_id=V3EHNOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637326708 (33h42m1.338919172s) [noise=true , address=77.90.31.17:4787 , piece_id=V3EHNOTNH6ZFVRWDL7WNLMR4NW3BFFGO3K5LJFIFVMRP6ROYFVIA, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637380935 (33h42m1.338973393s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1CrHsLV4vge154tJjMvbo5oxnphfukuXQBKfvTjA43n6CJM2Ah): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637394866 (33h42m1.338987321s) [noise=true , address=77.90.31.15:4709 , piece_id=2FDGKKQNZSCG244DCHV2LGWL23D2V2USRZEPPRTUUUDGQ4NPTEJQ, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637446506 (33h42m1.339038963s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MP1x3VuzX7cQPmD9pY3Cw1v6W2Q5LqUDdKesqX31vL9JLTRAp): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637456777 (33h42m1.339049231s) [noise=true , address=77.90.31.11:4581 , piece_id=MSHEXECZRQ7235NAOA7VUY2RA3UNPOI2UPDBCNDQJ2DMYHDH3GMA, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637522129 (33h42m1.339114607s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q3atbNPC93bXM7pr4KCeea5La9sX4xs2qNwGxNWSrNQs1ftrV): protocol: expected piece hash; context canceled; EOF
Oct 31 21:22:58.637528827 (33h42m1.339121275s) [noise=true , address=77.90.31.11:4573 , piece_id=QEIM2SKM3CI33HH25ZYJNIXWZBLNMOEGWPSYKO3BNMVQJSUYDEKQ, seg_pos={0 0}, upload=29647]: Long tail cancellation due to optimal threshold
Oct 31 21:22:58.637546736 (33h42m1.339139173s) [seg_pos={0 0}, upload=29647]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 31 21:22:58.637649415 (33h42m1.339241873s) [upload=29647]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 31 21:22:58.637657672 (33h42m1.339250145s) [upload=29647]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 31 21:22:58.787848722 (33h42m1.489441173s) [upload=29647]: Done uploading segments...
Oct 31 21:22:58.787868799 (33h42m1.489461251s) [upload=29647]: Upload finished. err: <nil>
Duration: 2.436656854s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
2.382628634s trace:
https://jaeger.dev.storj.io/trace/62df97e1df2d7a
FAIL (2025-10-31 21:25:39 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Oct 31 21:22:07.085611190 (33h41m9.787203641s) [upload=29636]: Starting upload
Oct 31 21:22:07.085688329 (33h41m9.787280767s) [upload=29636]: Done starting upload
Oct 31 21:22:07.085774737 (33h41m9.78736719s) [upload=29636]: Uploading segments...
Oct 31 21:22:07.086049239 (33h41m9.787641687s) [upload=29636]: Got next segment. Inline: false
Oct 31 21:22:07.086139291 (33h41m9.78773174s) [upload=29636]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 31 21:22:07.086145090 (33h41m9.787737538s) [upload=29636]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 31 21:22:07.198887636 (33h41m9.900480077s) [seg_pos={0 0}, upload=29636]: Begin upload segment...
Oct 31 21:22:07.198943347 (33h41m9.900535799s) [seg_pos={0 0}, upload=29636]: Done begin upload segment.
Oct 31 21:22:07.198976223 (33h41m9.900568674s) [noise=true , address=145.79.227.27:4689 , piece_id=E2674435DTU7SSPTAFGMRLBKZNPZAOIFMBDN5N734CP4RG23SVCA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199094043 (33h41m9.900686499s) [noise=true , address=145.79.227.15:4875 , piece_id=GXNV3O7D7URXRTLWVXMFMVTXGHZ4PAZMGQLI3L5SNUX3LDV7OXJA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199764319 (33h41m9.901356931s) [noise=true , address=145.79.227.14:4840 , piece_id=MKR2JQOAFU6BA6JUG4LTAICYEDRI367JLXCNW6EPZWIJR2ZNKXYA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.199979140 (33h41m9.901571602s) [noise=true , address=145.79.227.28:4725 , piece_id=2UEZO2DIIDOVE4V6TXLKZAWXTOUNTIJKCBOP6JHREEEYBUWEXAUA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.201094736 (33h41m9.902687196s) [noise=true , address=145.79.227.10:4401 , piece_id=YCZ6LL4ECE2QKCLEAF77L3C72VWCZ6IQVJLJ77RIACX4LIK2IJJA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.201150898 (33h41m9.902743361s) [noise=true , address=145.79.227.32:4834 , piece_id=LBXTDM2QLBWFNDJ5272FY546TCRWC3IFYRNOBCCIS3KQDFMVJDFQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.202300922 (33h41m9.903893383s) [noise=true , address=45.91.116.4:4940 , piece_id=7TW5URZLYHRH7MQX3PZGMYPKJS3SYTXCJJDSKCOQZHB2HIS6J4OA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.202347708 (33h41m9.903940165s) [noise=true , address=77.90.31.12:4590 , piece_id=WQQC3535ERLWQOD6WCWJF3DUY4FY4CZF7KPD6Y2K3Q2NYQCP45JA, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.203610241 (33h41m9.905202697s) [noise=true , address=145.79.227.31:4798 , piece_id=QIYHHYEZGJEMENOVRJIDH7722MX2HI7XAGWUSFKIV3YNHKVPVD2Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.204358358 (33h41m9.905950816s) [noise=true , address=145.79.227.9:4367 , piece_id=Q2S6IK6OE4535GKC6J55MV3N45EUDWGOYKWS3ISQSZJKW2WJCD6Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.204958129 (33h41m9.906550588s) [noise=true , address=45.91.116.4:4939 , piece_id=VYJO7TDD6SANMOZ7BB62KYLU72ACLUIZI7IHFV6I2PDAV7K7WLCQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.206076686 (33h41m9.907669138s) [noise=true , address=145.79.227.30:4763 , piece_id=ET72Q2LHTA3HQY2GQNCO2ZEOBP526EOWM72TL5XRQYOLLARFPZLQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.207450031 (33h41m9.909042495s) [noise=true , address=45.91.116.8:4045 , piece_id=B6C3QMJBWPBBJBTSYFP67R5WOVVGIGOYN6B6P5BAW35FXYEVPW3A, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.208460871 (33h41m9.910053312s) [noise=true , address=145.79.227.15:4873 , piece_id=FIIZHJVUTZ42TATHLHZEWSRMVUFB7DYTWGIABNR4LJQYVH4SKQ5A, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.209461830 (33h41m9.911054339s) [noise=true , address=145.79.227.11:4729 , piece_id=ZYPWL4SEIZHXYOAW4I5JOCPXDTHRCMO2FXYGA4ZUNHA2CSZ6DG5Q, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.210046720 (33h41m9.911639172s) [noise=true , address=145.79.227.30:4762 , piece_id=4BL5PJ3S2OC7PVSILA4IQCFT4JW6W5GBOXI7WRAS52DNBCF3DIHQ, seg_pos={0 0}, upload=29636]: Uploading piece...
Oct 31 21:22:07.808217579 (33h41m10.509810059s) [upload=29636]: Next returned nil segment
Oct 31 21:22:07.808236009 (33h41m10.509828464s) [upload=29636]: Waiting for error group managing segments...
Oct 31 21:22:07.951052914 (33h41m10.65264537s) [noise=true , address=145.79.227.32:4834 , piece_id=LBXTDM2QLBWFNDJ5272FY546TCRWC3IFYRNOBCCIS3KQDFMVJDFQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.951804679 (33h41m10.653397134s) [noise=true , address=145.79.227.15:4875 , piece_id=GXNV3O7D7URXRTLWVXMFMVTXGHZ4PAZMGQLI3L5SNUX3LDV7OXJA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.951868872 (33h41m10.653461324s) [noise=true , address=145.79.227.31:4798 , piece_id=QIYHHYEZGJEMENOVRJIDH7722MX2HI7XAGWUSFKIV3YNHKVPVD2Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952470935 (33h41m10.654063385s) [noise=true , address=145.79.227.30:4762 , piece_id=4BL5PJ3S2OC7PVSILA4IQCFT4JW6W5GBOXI7WRAS52DNBCF3DIHQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952500937 (33h41m10.654093387s) [noise=true , address=145.79.227.27:4689 , piece_id=E2674435DTU7SSPTAFGMRLBKZNPZAOIFMBDN5N734CP4RG23SVCA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.952980719 (33h41m10.654573168s) [noise=true , address=145.79.227.28:4725 , piece_id=2UEZO2DIIDOVE4V6TXLKZAWXTOUNTIJKCBOP6JHREEEYBUWEXAUA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.953254355 (33h41m10.654846814s) [noise=true , address=145.79.227.11:4729 , piece_id=ZYPWL4SEIZHXYOAW4I5JOCPXDTHRCMO2FXYGA4ZUNHA2CSZ6DG5Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.953604050 (33h41m10.6551965s) [noise=true , address=145.79.227.15:4873 , piece_id=FIIZHJVUTZ42TATHLHZEWSRMVUFB7DYTWGIABNR4LJQYVH4SKQ5A, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.954075643 (33h41m10.6556681s) [noise=true , address=145.79.227.9:4367 , piece_id=Q2S6IK6OE4535GKC6J55MV3N45EUDWGOYKWS3ISQSZJKW2WJCD6Q, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.958770427 (33h41m10.660362881s) [noise=true , address=145.79.227.10:4401 , piece_id=YCZ6LL4ECE2QKCLEAF77L3C72VWCZ6IQVJLJ77RIACX4LIK2IJJA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.965022314 (33h41m10.666614767s) [noise=true , address=145.79.227.30:4763 , piece_id=ET72Q2LHTA3HQY2GQNCO2ZEOBP526EOWM72TL5XRQYOLLARFPZLQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:07.976650683 (33h41m10.678243138s) [noise=true , address=145.79.227.14:4840 , piece_id=MKR2JQOAFU6BA6JUG4LTAICYEDRI367JLXCNW6EPZWIJR2ZNKXYA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: <nil>
Oct 31 21:22:36.799544661 (33h41m39.501137117s) [noise=true , address=45.91.116.4:4939 , piece_id=VYJO7TDD6SANMOZ7BB62KYLU72ACLUIZI7IHFV6I2PDAV7K7WLCQ, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection; write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:60134->45.91.116.4:4939: use of closed network connection
Oct 31 21:22:36.799605331 (33h41m39.50119779s) [noise=true , address=77.90.31.12:4590 , piece_id=WQQC3535ERLWQOD6WCWJF3DUY4FY4CZF7KPD6Y2K3Q2NYQCP45JA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection; write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:46910->77.90.31.12:4590: use of closed network connection
Oct 31 21:22:36.799747990 (33h41m39.501340451s) [noise=true , address=45.91.116.4:4940 , piece_id=7TW5URZLYHRH7MQX3PZGMYPKJS3SYTXCJJDSKCOQZHB2HIS6J4OA, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: read tcp 139.59.144.22:36232->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:36.799846589 (33h41m39.501439048s) [noise=true , address=45.91.116.8:4045 , piece_id=B6C3QMJBWPBBJBTSYFP67R5WOVVGIGOYN6B6P5BAW35FXYEVPW3A, seg_pos={0 0}, upload=29636]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: read tcp 139.59.144.22:39300->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 31 21:22:36.799901279 (33h41m39.501493715s) [seg_pos={0 0}, upload=29636]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Oct 31 21:22:36.906137907 (33h41m39.607730362s) [upload=29636]: Done uploading segments...
Oct 31 21:22:36.906171689 (33h41m39.60776414s) [upload=29636]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 2.382662369s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
6 out of 60 is failed (threshold=6) [..X....XX...................X..............X.........X......] duration:
1.689371519s trace:
https://jaeger.dev.storj.io/trace/5b5403da1e956ee2
FAIL (2025-10-31 21:25:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.689402329s
core
uplink
us-select-1
us1-placement-20
|