Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 190763 reports (Page 444 of 3816)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......X..............X.........X.......X...................] duration: 1.738409676s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:41:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.738433551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.........................XX.....XX.................] duration: 2.472954227s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:41:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.472992078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.....X.X...............] duration: 2.322346872s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:40:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.322369313s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X...............X...........X......XX................] duration: 2.707811335s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:40:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.707838421s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.881928443s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:40:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.882024988s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........X..............X.........X.......X..................] duration: 1.790118781s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:40:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.790140967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X.........................XX.....XX................] duration: 2.370660721s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:39:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.370691736s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.84629049s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:39:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.846312945s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X...............X...........X......XX...............] duration: 2.742778934s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:39:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.74313937s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.....X.X..............] duration: 2.404064143s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:39:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.404189841s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.........X..............X.........X.......X.................] duration: 1.681868155s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:39:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.681984697s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................................X.....X.X.............] duration: 2.584531028s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:39:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.584561664s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X.........................XX.....XX...............] duration: 2.409697122s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:39:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.40974248s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X...............X...........X......XX..............] duration: 2.733959623s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:38:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.733984218s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X.....X.X............] duration: 2.410727279s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:38:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.410770232s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.865367817s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:38:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.86539821s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X...............X...........X......XX.............] duration: 2.629441006s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:38:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.629482318s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X.........................XX.....XX..............] duration: 2.389725802s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:38:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.389752801s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..........X..............X.........X.......X................] duration: 1.685205569s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:38:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.685455712s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X.........................XX.....XX.............] duration: 2.356203896s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:38:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.356259796s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X...............X...........X......XX............] duration: 2.627607099s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:38:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.62765936s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...........X..............X.........X.......X...............] duration: 1.581606889s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:37:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.581628703s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................................X.....X.X...........] duration: 2.387508971s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:37:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.387629137s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X.........................XX.....XX............] duration: 2.304365163s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:37:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.304397368s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.709532887s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:37:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.709544339s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............X..............X.........X.......X..............] duration: 1.623138655s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:37:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.623153806s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X.....X.X..........] duration: 2.40294357s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:37:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.402979224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X.........................XX.....XX...........] duration: 2.365123477s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:36:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.365276649s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............X...............X...........X......XX...........] duration: 2.728085268s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:36:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.7281772s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X.........................XX.....XX..........] duration: 2.391051425s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:36:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.391080619s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 3.08870868s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:36:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.088739793s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............X..............X.........X.......X.............] duration: 1.59539832s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:36:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.59542568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X.....X.X.........] duration: 2.314832294s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:36:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.314859007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [................X.........................XX.....XX.........] duration: 2.441446039s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:36:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.441496173s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 3.046826441s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:36:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.046856644s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............X..............X.........X.......X............] duration: 1.705290064s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:35:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.70531386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............X...............X...........X......XX..........] duration: 2.704658735s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:35:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.704685572s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.773151476s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:35:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.773179528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.................X.........................XX.....XX........] duration: 2.287400431s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:35:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.287430081s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X.....X.X........] duration: 2.338416108s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:35:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.338452622s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...............X..............X.........X.......X...........] duration: 1.689320344s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:35:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.689363029s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............X...............X...........X......XX.........] duration: 2.646104317s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:34:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.646128596s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..................X.........................XX.....XX.......] duration: 2.359864897s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:34:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.359897007s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.817866289s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:34:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.817890648s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X.....X.X.......] duration: 2.464754722s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:34:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.464779959s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [................X..............X.........X.......X..........] duration: 1.763073114s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:34:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.763098593s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.903854644s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:33:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.903912924s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.................X..............X.........X.......X.........] duration: 1.678868549s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:33:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.678896711s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...............X...............X...........X......XX........] duration: 2.750032838s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:33:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.750562172s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X.....X.X......] duration: 2.435930234s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:33:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.435969295s core uplink us-select-1

1
...
442
443
444
445
446
...
3816