Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 190703 reports (Page 442 of 3815)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X...........X......XX.........................] duration: 2.490553614s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:46:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.490576296s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X.....X.X.......................] duration: 2.541371507s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:46:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.541387177s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 3.202604687s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:46:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.202637853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.........................XX.....XX........................] duration: 2.411633956s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:46:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.411662488s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X..............X.........X.......X..........................] duration: 2.092033514s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:46:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.092176063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.........................XX.....XX.......................] duration: 2.363571161s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:46:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.363600951s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X..............X.........X.......X.........................] duration: 1.645754883s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:46:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.645799167s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X...........X......XX........................] duration: 2.629646373s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:46:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.629686504s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X.....X.X......................] duration: 2.400202616s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:45:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.400225894s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X...............X...........X......XX.......................] duration: 2.562445446s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:45:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.562474949s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.930271584s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:45:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.930329778s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X..............X.........X.......X........................] duration: 1.658829582s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:45:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.658867437s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.........................XX.....XX......................] duration: 2.418143087s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:45:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.418169656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 3.737913174s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:45:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.737942248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X.....X.X.....................] duration: 2.413296684s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:45:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.413325385s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X...............X...........X......XX......................] duration: 2.623985331s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:44:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.624017195s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.........................XX.....XX.....................] duration: 2.414081325s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:44:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.414110853s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X..............X.........X.......X.......................] duration: 1.652063852s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:44:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.652092807s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.819688872s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:44:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.819715044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X.....X.X....................] duration: 2.520318578s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:44:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.520405004s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X...............X...........X......XX.....................] duration: 2.842575456s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:44:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.84260758s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X.....X.X...................] duration: 2.390848476s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:43:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.390870597s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X..............X.........X.......X......................] duration: 1.706210385s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:43:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.706233978s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X...............X...........X......XX....................] duration: 2.830391197s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:43:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.830419382s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X.....X.X..................] duration: 2.352269881s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:43:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.352297793s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.........................XX.....XX....................] duration: 2.332028823s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:43:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.332071585s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.974835792s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:43:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.97486272s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X...............X...........X......XX...................] duration: 2.706797393s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:42:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.70682709s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X..............X.........X.......X.....................] duration: 1.704371382s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:42:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.704415972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.........................XX.....XX...................] duration: 2.25784086s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:42:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.257869567s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.94999953s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:42:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.950027662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......X..............X.........X.......X....................] duration: 2.280635886s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:42:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.280661277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................X.....X.X.................] duration: 2.380566313s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:42:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.380843826s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.........................XX.....XX..................] duration: 2.325775431s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:42:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.325802916s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X...............X...........X......XX..................] duration: 2.690804423s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:41:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.690833521s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.94387459s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:41:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.943942864s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................X.....X.X................] duration: 2.390882423s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:41:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.390907514s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.940939605s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:41:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.940974914s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X...............X...........X......XX.................] duration: 2.526675654s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:41:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.526788134s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.950331233s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:41:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.950368096s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......X..............X.........X.......X...................] duration: 1.738409676s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:41:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.738433551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X.........................XX.....XX.................] duration: 2.472954227s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:41:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.472992078s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................X.....X.X...............] duration: 2.322346872s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:40:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.322369313s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X...............X...........X......XX................] duration: 2.707811335s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:40:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.707838421s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.881928443s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:40:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.882024988s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [........X..............X.........X.......X..................] duration: 1.790118781s trace: https://jaeger.dev.storj.io/trace/d59b5ba8bd72bed FAIL (2025-10-31 21:40:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.790140967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X.........................XX.....XX................] duration: 2.370660721s trace: https://jaeger.dev.storj.io/trace/47742a9fc7eef47b FAIL (2025-10-31 21:39:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 53281825s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252144150 (33h47m33.953736606s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252248964 (33h47m33.95384142s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.252760677 (33h47m33.954353125s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253071842 (33h47m33.954664299s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253216278 (33h47m33.954808727s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253451946 (33h47m33.955044403s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.253652570 (33h47m33.955245024s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254056115 (33h47m33.955648572s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254143198 (33h47m33.955735647s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254486294 (33h47m33.956078748s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.254602493 (33h47m33.956194944s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.255075289 (33h47m33.956667737s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Uploading piece... Oct 31 21:28:31.811752904 (33h47m34.513345347s) [upload=29727]: Next returned nil segment Oct 31 21:28:31.811768146 (33h47m34.513360595s) [upload=29727]: Waiting for error group managing segments... Oct 31 21:28:31.951788186 (33h47m34.653380635s) [noise=true , address=145.79.227.17:4003 , piece_id=33ZZHY6MVPRQQPXLHX4PIN7HAYHKO3TDMMGCPNZMAUGY2I4WJTXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.970243289 (33h47m34.67183574s) [noise=true , address=145.79.227.50:4400 , piece_id=IMMIBWNJAUZYEO5LVOHKEHWM4RPSLAKIBN4YCYUQOIN7YNMU4SYQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.981809361 (33h47m34.683401813s) [noise=true , address=145.79.227.38:4995 , piece_id=46UA2OSAKSHGE5SAP5RRWOJBM3BBU2DRS4HFWMNLUDV7EY2XGPGQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.985828089 (33h47m34.687420544s) [noise=true , address=145.79.227.21:4483 , piece_id=MQ7S4LMHDJHYBFW5EQK76ZZ6L2LRQWLO2U6NCTBHZCIQE4JWPJAQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:31.998128117 (33h47m34.699720581s) [noise=true , address=145.79.227.3:4172 , piece_id=ATNJUHZMEEXIEQGDILYSJ5AFJROJEVZUNVPUGH3SJIYC65F455BA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.003077819 (33h47m34.704670289s) [noise=true , address=145.79.227.20:4445 , piece_id=GF755LF2ZHS74L5IABAXBVT74ZZHBADBSDSAVJKSICH4MVIW34SA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.847776743 (33h47m35.549369201s) [noise=true , address=145.79.227.16:4940 , piece_id=XMU27VSPYG3UEF3PWHPLL3G5COVQUCNEQCX6JAEE65CW7CISFCLA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:32.910129540 (33h47m35.611721999s) [noise=true , address=45.91.116.6:4983 , piece_id=BHFPTUCWNIO5JXUVSHSZZH43E643YF6JZPIIISSMS5M5YKSPFHQQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.037396899 (33h47m35.738989347s) [noise=true , address=45.91.116.11:4127 , piece_id=EPVCGGZS7YREJRIVLWYDD3YIC6Q66M22RGIJ5XDJEFES7A2VADWQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.575136010 (33h47m36.276728458s) [noise=true , address=45.91.116.7:4027 , piece_id=YU4BEPUJ26VZABUZ72SQOMTRP4SIAMUKUBU2YSN7NH7XXH2LMBUQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.580271724 (33h47m36.281864172s) [noise=true , address=77.90.31.12:4600 , piece_id=3F77YW2BIS7UYO53ZMA4JRH4GCV4LUL7RV6ECFR2GV6OZGYOQZZQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.721191718 (33h47m36.422784177s) [noise=true , address=45.91.116.13:4186 , piece_id=Q3FC3SSVDQAK2HB46BCLLEEIVATWL37275EEWU4IC4ZGYQPVEZXQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:33.937166015 (33h47m36.638758463s) [noise=true , address=77.90.31.16:4751 , piece_id=XLXXB6X6OYDJ54YTIP3EHE76F6A5FSY7C2UQX2OEI32JKGSE6TRA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095755299 (33h47m36.797347752s) [noise=true , address=45.91.116.10:4107 , piece_id=K3VUQI7LZ4IS45NVJTV5KG43NJ22C2X6PQPG5Q3FCNAM467GWMFA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: <nil> Oct 31 21:28:34.095786066 (33h47m36.797378521s) [seg_pos={0 0}, upload=29727]: Segment reached optimal threshold of 14 pieces. Oct 31 21:28:34.096062598 (33h47m36.797655056s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKhKeEpeFSQDeKBUkSwGp58G9nAcdQZ19zuuuae522vMv7AP9): protocol: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:32826->77.90.31.11:4564: use of closed network connection Oct 31 21:28:34.096072137 (33h47m36.797664588s) [noise=true , address=77.90.31.11:4564 , piece_id=NF7N273APLAAHW2OH2SE4UY5RNXNO5DGKQY2IN2GRPX5EJD3UTWA, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096481415 (33h47m36.798073874s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NKFSj1P8HBhfYrYPTsj5hANnpp3FZmMkuWvqpS7MxKR3k6Bq): protocol: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45774->77.90.31.10:4526: use of closed network connection Oct 31 21:28:34.096501202 (33h47m36.79809366s) [noise=true , address=77.90.31.10:4526 , piece_id=G4F4CA36DLM7KT67VLU2BEESAGNN4DORQN6QTXV7K42R772HXT4Q, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096719876 (33h47m36.798312335s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NgPjVCyZZAfxVr3WvKrGRjnkpPtUvmQcCMiYntxNKVSWqXcHC): protocol: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:47680->77.90.31.21:4893: use of closed network connection Oct 31 21:28:34.096728272 (33h47m36.79832073s) [noise=true , address=77.90.31.21:4893 , piece_id=JV57ZNQUURS2TWZZEKF3AWBOQ2FMQ4NYODY2ZWEETD7BSCPSJZ2A, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096895837 (33h47m36.798488295s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1h41MzKfNcdZxHpUtH1MiybVcSxAYPNNwnmSELZieiUW2fY2kw): protocol: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53468->77.90.31.15:4705: use of closed network connection Oct 31 21:28:34.096903336 (33h47m36.798495793s) [noise=true , address=77.90.31.15:4705 , piece_id=FVFSPVGBU3CEQQ3SSIFJY5EBETRZ24TLOBAGTAN4SXF2QVYSKZLQ, seg_pos={0 0}, upload=29727]: Long tail cancellation due to optimal threshold Oct 31 21:28:34.096933660 (33h47m36.798526135s) [seg_pos={0 0}, upload=29727]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 31 21:28:34.097042362 (33h47m36.798634814s) [upload=29727]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:34.097050481 (33h47m36.798642934s) [upload=29727]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:34.235018068 (33h47m36.936610525s) [upload=29727]: Done uploading segments... Oct 31 21:28:34.235041482 (33h47m36.936633943s) [upload=29727]: Upload finished. err: <nil> Duration: 2.370691736s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.84629049s trace: https://jaeger.dev.storj.io/trace/563abad3e0219f6f FAIL (2025-10-31 21:39:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.846312945s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X...............X...........X......XX...............] duration: 2.742778934s trace: https://jaeger.dev.storj.io/trace/36ef0ed973c25d7b FAIL (2025-10-31 21:39:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.74313937s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X.....X.X..............] duration: 2.404064143s trace: https://jaeger.dev.storj.io/trace/4240f607f15f8845 FAIL (2025-10-31 21:39:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q) context deadline exceeded ======LOGS====== load=29731]: Done starting upload Oct 31 21:28:36.728396910 (33h47m39.429989358s) [upload=29731]: Uploading segments... Oct 31 21:28:36.728446515 (33h47m39.430038963s) [upload=29731]: Got next segment. Inline: false Oct 31 21:28:36.728553815 (33h47m39.430146263s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 31 21:28:36.728714698 (33h47m39.430307146s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 31 21:28:36.842679192 (33h47m39.544271668s) [seg_pos={0 0}, upload=29731]: Begin upload segment... Oct 31 21:28:36.842756500 (33h47m39.544348953s) [seg_pos={0 0}, upload=29731]: Done begin upload segment. Oct 31 21:28:36.842819979 (33h47m39.544412436s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.844481939 (33h47m39.546074402s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845505034 (33h47m39.547097492s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.845599074 (33h47m39.547191533s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846582945 (33h47m39.548175404s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.846738734 (33h47m39.548331191s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847566779 (33h47m39.549159237s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.847745957 (33h47m39.54933841s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.848365430 (33h47m39.549957889s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.849327153 (33h47m39.550919624s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850109778 (33h47m39.551702233s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.850790717 (33h47m39.552383177s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.851473602 (33h47m39.553066055s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852085173 (33h47m39.553677631s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.852636775 (33h47m39.554229226s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:36.853291802 (33h47m39.55488426s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Uploading piece... Oct 31 21:28:37.463649505 (33h47m40.165241985s) [upload=29731]: Next returned nil segment Oct 31 21:28:37.463729471 (33h47m40.165321942s) [upload=29731]: Waiting for error group managing segments... Oct 31 21:28:37.602639666 (33h47m40.304232117s) [noise=true , address=145.79.227.13:4804 , piece_id=FMQNM65RQOPUL2I7QJHWSVJ26HRUZZJNJUETREQX5HH4RZR5BJAA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.603060455 (33h47m40.304652903s) [noise=true , address=145.79.227.27:4691 , piece_id=N7R56I3P5ADXYMNWJ5OOBP5UIC636WWBXWERU5DV2FKPAI4OAJ7A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.604255592 (33h47m40.305848042s) [noise=true , address=145.79.227.27:4690 , piece_id=HHUINKJFS5UPCNWNV5GQZGF4I6JXXUZSWOZGSXTQPHL3DJCYZ73Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.607084649 (33h47m40.308677087s) [noise=true , address=145.79.227.31:4798 , piece_id=CVE7W2Y2X2BMPYNVPDIH7OIYS635DJ7PMZLZM4MVPTVOCZS3MD6Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.608327378 (33h47m40.309919832s) [noise=true , address=145.79.227.30:4763 , piece_id=FHTPKAXVMHMB53YXVUM3V2UCGTY5VHAUZOAC25MB5ZELCHINZEIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609249623 (33h47m40.310842071s) [noise=true , address=145.79.227.25:4618 , piece_id=FI2HOBX7EASWQDRMXBY6NB2NLQN34BCCSLU4AR5RRPHFPQN6PS3A, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.609555784 (33h47m40.311148236s) [noise=true , address=145.79.227.9:4366 , piece_id=3X6TWU3MUQSUY3AASXQS2B7TOXBCLF4R5JQ4BSLWLCSDT3PW5NUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:37.615075256 (33h47m40.316667706s) [noise=true , address=145.79.227.23:4546 , piece_id=R2Q2KV3Z6CEFXEHSRR7PFCACVE667NENYZRF4FK35M6ZFG73FITQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:38.840000720 (33h47m41.541593168s) [noise=true , address=45.91.116.9:4263 , piece_id=ET753RUALNX65W3DXVVQ2QHB6RUE7LKCSDDXTRKNN45WT7EBXSYA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:39.338120479 (33h47m42.039712937s) [noise=true , address=77.90.31.10:4520 , piece_id=SHGKHM5ES3IQHFWPUBEPKSDLJF5VNLPQU5I2JA2TIBPCIQM7TKKA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:40.199129684 (33h47m42.900722135s) [noise=true , address=77.90.31.18:4805 , piece_id=QJYRW47ALK2U2D5DXAZ5CWJ72ULV2S5ASUITFQ2P67ZXMEOJC6RA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:41.244230004 (33h47m43.945822453s) [noise=true , address=45.91.116.8:4045 , piece_id=SX46VM63PYNBHUORJMHBJHIRB56S774RYPM4KQ75OCXG2XAUIJ5Q, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411275317 (33h47m56.112867766s) [noise=true , address=77.90.31.11:4555 , piece_id=7HZWIFKCGVIB5KQAVVSN2X2T2SP3W7ZKNKEEMK7TSHGE7DC7TMIQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: <nil> Oct 31 21:28:53.411298071 (33h47m56.112890523s) [seg_pos={0 0}, upload=29731]: Segment reached optimal threshold of 13 pieces. Oct 31 21:28:53.411554553 (33h47m56.113147006s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:45006->77.90.31.19:4841: use of closed network connection Oct 31 21:28:53.411563265 (33h47m56.11315572s) [noise=true , address=77.90.31.19:4841 , piece_id=YB7RED5YLJ7TJKHLZ2CRKFV2H4XRCIIAI2E55JVXNMW3GV5JQEDQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.411668853 (33h47m56.113261309s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ): protocol: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:53598->77.90.31.21:4879: use of closed network connection Oct 31 21:28:53.411705207 (33h47m56.113297658s) [noise=true , address=77.90.31.21:4879 , piece_id=XWVA6D4N6JM75HMGJJ264TCVYC76INARQLT3OUB4Q7QSZ7KVWRUA, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412001982 (33h47m56.113594436s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: expected piece hash; context canceled; EOF Oct 31 21:28:53.412025199 (33h47m56.113617648s) [noise=true , address=77.90.31.14:4662 , piece_id=DN47CYYSJ4LKS44U2ZFQV5O63UGD3XB62MNXTWLSSGLFZQ3WHQJQ, seg_pos={0 0}, upload=29731]: Long tail cancellation due to optimal threshold Oct 31 21:28:53.412047295 (33h47m56.113639732s) [seg_pos={0 0}, upload=29731]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 31 21:28:53.412162254 (33h47m56.113754702s) [upload=29731]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 31 21:28:53.412181460 (33h47m56.113773909s) [upload=29731]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 31 21:28:53.558308553 (33h47m56.259901015s) [upload=29731]: Done uploading segments... Oct 31 21:28:53.558331822 (33h47m56.259924661s) [upload=29731]: Upload finished. err: <nil> Duration: 2.404189841s core uplink us-select-1

1
...
440
441
442
443
444
...
3815