Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 29 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000710856s FAIL (2025-08-07 21:34:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000725151s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..............................................XXXXXXXXXXXXXX] duration: 30.000539435s FAIL (2025-08-07 21:34:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ]: Uploading piece... Aug 7 21:20:13.799847282 (11h40m35.543904151s) [noise=true , address=77.90.31.13:4626 , piece_id=VNGCTMK7ET2RCVPEDCDXZHYKARZIIGUU6CPVT6LKA7HCLWJFJ7AQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.800536275 (11h40m35.544593146s) [noise=true , address=45.91.116.5:4297 , piece_id=77LOHOMR6FRLTTYGTAVC4SHMSDSDWGNITJMIIKDIA7ADS2272TAQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.800663553 (11h40m35.544720423s) [noise=true , address=45.91.116.12:4226 , piece_id=47PAAFR7VRN7H7MBS7Z7KRONLGN6Q2JF5CBYT6ASUIANYBUSYNKQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:14.404668625 (11h40m36.148725475s) [upload=12096]: Next returned nil segment Aug 7 21:20:14.404685815 (11h40m36.148742676s) [upload=12096]: Waiting for error group managing segments... Aug 7 21:20:14.561430060 (11h40m36.305486911s) [noise=true , address=45.91.116.12:4225 , piece_id=EHBNQPDYM7HB4OGRJGU6QZ3CTM5YWHUEV2SSD5LKUZI6LGG6X4GA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.583124069 (11h40m36.327180919s) [noise=true , address=104.36.82.15:4257 , piece_id=46HGSNBRDPF3JZMTPWY3OPAGK3RXBGTZWXDDNWZQERV6QAYWJ3TA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.622563656 (11h40m36.366620519s) [noise=true , address=45.91.116.8:4047 , piece_id=RCBP5UBGZ5C6YEQQQ3SD3GUU26LJTFX3GOL4XZOKEFZOXB4WDRSA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.648162660 (11h40m36.392219508s) [noise=true , address=77.90.31.17:4772 , piece_id=EMV6OLIN7YSBQQN4SC4VJGWQGFKLPBQVVJZTTKUE5ELQDXMXPSSQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.649278924 (11h40m36.393335781s) [noise=true , address=104.36.82.7:4912 , piece_id=2XMUVIQB6QCEDOJGJR7CY6FW6PALYZMQRA7KNPSQQB4IAYHPQZRA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.657067139 (11h40m36.401123988s) [noise=true , address=77.90.31.18:4807 , piece_id=O3SDUB6BQB6CPYN4X7Z7GHUK5Y663IOBGE4S4MB5UBB46HCQMJLA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.660665476 (11h40m36.404722337s) [noise=true , address=77.90.31.13:4627 , piece_id=TOKP44CMWL4OXYLWOFHL7WFKSN2C57WGHWBPZMQDVZ4GRDKLCBRA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.692922160 (11h40m36.43697902s) [noise=true , address=104.36.82.2:4731 , piece_id=DKWMN2CLFT3XVYTGEF3HJYZXPME5TE37I45TD5ABWVRQMWE3X6YA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.693482368 (11h40m36.437539243s) [noise=true , address=45.91.116.12:4226 , piece_id=47PAAFR7VRN7H7MBS7Z7KRONLGN6Q2JF5CBYT6ASUIANYBUSYNKQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.722004664 (11h40m36.466061524s) [noise=true , address=45.91.116.5:4297 , piece_id=77LOHOMR6FRLTTYGTAVC4SHMSDSDWGNITJMIIKDIA7ADS2272TAQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.747287536 (11h40m36.491344386s) [noise=true , address=45.91.116.6:4973 , piece_id=UKM3AZMNW67WQ7WMFP5Z55NG4JBZBFLDVODUHIISQYZ27HUVHP6Q, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.751701687 (11h40m36.495758555s) [noise=true , address=77.90.31.18:4805 , piece_id=HKKJRNDDMITSXEXDDMZ37ZCG5OFNG72PAEDN7TNWDHXZQYBSWIXA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.798490539 (11h40m36.542547386s) [noise=true , address=77.90.31.10:4519 , piece_id=FDV33ZMCVYJOMT4YP7KIRA3MBB7OUNJHAPITYYLMXJW6DZWKGGCQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.798534450 (11h40m36.54259131s) [seg_pos={0 0}, upload=12096]: Segment reached optimal threshold of 13 pieces. Aug 7 21:20:14.798673827 (11h40m36.542730687s) [noise=true , address=45.91.116.10:4082 , piece_id=3AL7IMQKZU26B34QQEK43SAXSWT6W7M4TF4ITTYF774U2LED2LRQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798750267 (11h40m36.542807123s) [noise=true , address=77.90.31.13:4626 , piece_id=VNGCTMK7ET2RCVPEDCDXZHYKARZIIGUU6CPVT6LKA7HCLWJFJ7AQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798798579 (11h40m36.542855426s) [noise=true , address=45.91.116.4:4937 , piece_id=PGZP47Q6KAXPGOPJ5JAAKADDBRWMAIBX5GWL6JGHFJUU3AMOPKTQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798820360 (11h40m36.54287721s) [seg_pos={0 0}, upload=12096]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:20:14.798910105 (11h40m36.542966966s) [upload=12096]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:20:14.798915686 (11h40m36.542972547s) [upload=12096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:24.464709964 (11h51m46.208766834s) [upload=12096]: Done uploading segments... Aug 7 21:31:24.464745783 (11h51m46.208802653s) [upload=12096]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:24.881904902 (11h51m46.625961766s) [upload=12268]: Starting upload Aug 7 21:31:24.882094151 (11h51m46.626151016s) [upload=12268]: Done starting upload Aug 7 21:31:24.882164223 (11h51m46.626221082s) [upload=12268]: Uploading segments... Aug 7 21:31:24.882475408 (11h51m46.626532277s) [upload=12268]: Got next segment. Inline: false Aug 7 21:31:24.882552261 (11h51m46.626609127s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:24.882560165 (11h51m46.626617032s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:54.614102730 (11h52m16.358159588s) [upload=12268]: Next segment err: context deadline exceeded Aug 7 21:31:54.614205863 (11h52m16.358262714s) [upload=12268]: Done uploading segments... Aug 7 21:31:54.614225924 (11h52m16.358282773s) [upload=12268]: Upload finished. err: context deadline exceeded Aug 7 21:32:00.253568814 (11h52m21.997625667s) [upload=12278]: Starting upload Aug 7 21:32:00.253669214 (11h52m21.997726066s) [upload=12278]: Done starting upload Aug 7 21:32:00.254008956 (11h52m21.998065808s) [upload=12278]: Uploading segments... Aug 7 21:32:00.254068254 (11h52m21.998125125s) [upload=12278]: Got next segment. Inline: false Aug 7 21:32:00.254112779 (11h52m21.998169639s) [upload=12278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:00.254118157 (11h52m21.998175018s) [upload=12278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:29.984781139 (11h52m51.728838008s) [upload=12278]: Next segment err: context deadline exceeded Aug 7 21:32:29.984827795 (11h52m51.728884645s) [upload=12278]: Done uploading segments... Aug 7 21:32:29.984851910 (11h52m51.728908763s) [upload=12278]: Upload finished. err: context deadline exceeded Aug 7 21:32:35.321237619 (11h52m57.065294469s) [upload=12287]: Starting upload Aug 7 21:32:35.321310476 (11h52m57.065367326s) [upload=12287]: Done starting upload Aug 7 21:32:35.321418401 (11h52m57.06547525s) [upload=12287]: Uploading segments... Aug 7 21:32:35.321610690 (11h52m57.06566754s) [upload=12287]: Got next segment. Inline: false Aug 7 21:32:35.321654358 (11h52m57.065711209s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:35.321659987 (11h52m57.065716847s) [upload=12287]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:05.046284716 (11h53m26.790341587s) [upload=12287]: Next segment err: context deadline exceeded Aug 7 21:33:05.046403894 (11h53m26.790460755s) [upload=12287]: Done uploading segments... Aug 7 21:33:05.046426465 (11h53m26.790483318s) [upload=12287]: Upload finished. err: context deadline exceeded Aug 7 21:33:21.055953150 (11h53m42.80001s) [upload=12296]: Starting upload Aug 7 21:33:21.056022172 (11h53m42.800079036s) [upload=12296]: Done starting upload Aug 7 21:33:21.056266284 (11h53m42.800323139s) [upload=12296]: Uploading segments... Aug 7 21:33:21.056360175 (11h53m42.800417035s) [upload=12296]: Got next segment. Inline: false Aug 7 21:33:21.056419681 (11h53m42.800476544s) [upload=12296]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:21.056426657 (11h53m42.800483517s) [upload=12296]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:50.766946667 (11h54m12.511003536s) [upload=12296]: Next segment err: context deadline exceeded Aug 7 21:33:50.767114018 (11h54m12.511170869s) [upload=12296]: Done uploading segments... Aug 7 21:33:50.767133351 (11h54m12.511190205s) [upload=12296]: Upload finished. err: context deadline exceeded Aug 7 21:34:24.125593602 (11h54m45.869650451s) [upload=12312]: Starting upload Aug 7 21:34:24.125684240 (11h54m45.869741107s) [upload=12312]: Done starting upload Aug 7 21:34:24.125788426 (11h54m45.86984528s) [upload=12312]: Uploading segments... Aug 7 21:34:24.126065319 (11h54m45.870122183s) [upload=12312]: Got next segment. Inline: false Aug 7 21:34:24.126124945 (11h54m45.870181799s) [upload=12312]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:24.126132194 (11h54m45.870189062s) [upload=12312]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:53.850914976 (11h55m15.594971842s) [upload=12312]: Next segment err: context deadline exceeded Aug 7 21:34:53.851009806 (11h55m15.595066658s) [upload=12312]: Done uploading segments... Aug 7 21:34:53.851029858 (11h55m15.595086713s) [upload=12312]: Upload finished. err: context deadline exceeded Duration: 30.000580715s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.............................X..XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001438856s FAIL (2025-08-07 21:34:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001473822s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 30.001180439s FAIL (2025-08-07 21:34:41 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001214367s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 1m0.000747639s FAIL (2025-08-07 21:34:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000768994s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 967.658243ms FAIL (2025-08-07 21:34:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== loading segments... Aug 7 21:21:33.493714811 (11h41m55.237771657s) [upload=12125]: Upload finished. err: <nil> Aug 7 21:22:47.998800022 (11h43m9.742856869s) [upload=12141]: Starting upload Aug 7 21:22:47.998849424 (11h43m9.742906283s) [upload=12141]: Done starting upload Aug 7 21:22:47.998919383 (11h43m9.742976233s) [upload=12141]: Uploading segments... Aug 7 21:22:47.999001069 (11h43m9.743057938s) [upload=12141]: Got next segment. Inline: true Aug 7 21:22:47.999007866 (11h43m9.743064718s) [upload=12141]: Waiting for error group managing segments... Aug 7 21:22:47.999066684 (11h43m9.743123533s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:47.999073371 (11h43m9.743130221s) [upload=12141]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:47.999077096 (11h43m9.743133947s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:48.164602559 (11h43m9.90865943s) [upload=12141]: Done uploading segments... Aug 7 21:22:48.164642998 (11h43m9.908699853s) [upload=12141]: Upload finished. err: <nil> Aug 7 21:23:35.990065388 (11h43m57.734122239s) [upload=12154]: Starting upload Aug 7 21:23:35.990143300 (11h43m57.734200146s) [upload=12154]: Done starting upload Aug 7 21:23:35.990217262 (11h43m57.734274108s) [upload=12154]: Uploading segments... Aug 7 21:23:35.990277036 (11h43m57.734333925s) [upload=12154]: Got next segment. Inline: true Aug 7 21:23:35.990283708 (11h43m57.734340569s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:23:35.990337388 (11h43m57.734394253s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:35.990343849 (11h43m57.734400714s) [upload=12154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:35.990347914 (11h43m57.734405001s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:36.150463692 (11h43m57.894520544s) [upload=12154]: Done uploading segments... Aug 7 21:23:36.150483857 (11h43m57.894540707s) [upload=12154]: Upload finished. err: <nil> Aug 7 21:24:13.830997960 (11h44m35.57505482s) [upload=12160]: Starting upload Aug 7 21:24:13.831049433 (11h44m35.575106295s) [upload=12160]: Done starting upload Aug 7 21:24:13.831151824 (11h44m35.575208678s) [upload=12160]: Uploading segments... Aug 7 21:24:13.831192510 (11h44m35.575249377s) [upload=12160]: Got next segment. Inline: true Aug 7 21:24:13.831197614 (11h44m35.575254475s) [upload=12160]: Waiting for error group managing segments... Aug 7 21:24:13.831253497 (11h44m35.57531036s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:13.831259191 (11h44m35.575316046s) [upload=12160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:13.831262688 (11h44m35.575319551s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:13.982951222 (11h44m35.727008092s) [upload=12160]: Done uploading segments... Aug 7 21:24:13.982975878 (11h44m35.727032727s) [upload=12160]: Upload finished. err: <nil> Aug 7 21:24:48.198960455 (11h45m9.943017328s) [upload=12172]: Starting upload Aug 7 21:24:48.199050501 (11h45m9.943107348s) [upload=12172]: Done starting upload Aug 7 21:24:48.199166478 (11h45m9.943223341s) [upload=12172]: Uploading segments... Aug 7 21:24:48.199284289 (11h45m9.943341159s) [upload=12172]: Got next segment. Inline: true Aug 7 21:24:48.199309162 (11h45m9.943366036s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:24:48.199360469 (11h45m9.943417336s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:48.199382302 (11h45m9.943439163s) [upload=12172]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:48.199386864 (11h45m9.943443734s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:48.353366598 (11h45m10.09742347s) [upload=12172]: Done uploading segments... Aug 7 21:24:48.353410465 (11h45m10.097467327s) [upload=12172]: Upload finished. err: <nil> Aug 7 21:25:56.601363668 (11h46m18.345420518s) [upload=12190]: Starting upload Aug 7 21:25:56.601416202 (11h46m18.345473063s) [upload=12190]: Done starting upload Aug 7 21:25:56.601478987 (11h46m18.345535837s) [upload=12190]: Uploading segments... Aug 7 21:25:56.601519567 (11h46m18.345576436s) [upload=12190]: Got next segment. Inline: true Aug 7 21:25:56.601524613 (11h46m18.345581463s) [upload=12190]: Waiting for error group managing segments... Aug 7 21:25:56.601564530 (11h46m18.345621381s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:56.601571562 (11h46m18.345628412s) [upload=12190]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:56.601574952 (11h46m18.345631806s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.289489478 (11h46m19.033546339s) [upload=12190]: Done uploading segments... Aug 7 21:25:57.289514011 (11h46m19.033570861s) [upload=12190]: Upload finished. err: <nil> Aug 7 21:26:32.814119549 (11h46m54.558176399s) [upload=12198]: Starting upload Aug 7 21:26:32.814178314 (11h46m54.558235164s) [upload=12198]: Done starting upload Aug 7 21:26:32.814230962 (11h46m54.558287814s) [upload=12198]: Uploading segments... Aug 7 21:26:32.814314990 (11h46m54.558371858s) [upload=12198]: Got next segment. Inline: true Aug 7 21:26:32.814321493 (11h46m54.558378353s) [upload=12198]: Waiting for error group managing segments... Aug 7 21:26:32.814387101 (11h46m54.558443951s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.814393685 (11h46m54.558450545s) [upload=12198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:32.814397318 (11h46m54.558454179s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:32.972796167 (11h46m54.716853027s) [upload=12198]: Done uploading segments... Aug 7 21:26:32.972815783 (11h46m54.716872632s) [upload=12198]: Upload finished. err: <nil> Aug 7 21:27:41.280232963 (11h48m3.024289826s) [upload=12214]: Starting upload Aug 7 21:27:41.280291948 (11h48m3.024348811s) [upload=12214]: Done starting upload Aug 7 21:27:41.280410716 (11h48m3.024467565s) [upload=12214]: Uploading segments... Aug 7 21:27:41.280478626 (11h48m3.024535487s) [upload=12214]: Got next segment. Inline: true Aug 7 21:27:41.280485326 (11h48m3.024542193s) [upload=12214]: Waiting for error group managing segments... Aug 7 21:27:41.280533304 (11h48m3.024590156s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:41.280538628 (11h48m3.024595488s) [upload=12214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:41.280542274 (11h48m3.02459914s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:41.433978896 (11h48m3.178035773s) [upload=12214]: Done uploading segments... Aug 7 21:27:41.434003733 (11h48m3.178060594s) [upload=12214]: Upload finished. err: <nil> Aug 7 21:28:53.851663561 (11h49m15.595720412s) [upload=12231]: Starting upload Aug 7 21:28:53.851736366 (11h49m15.595793226s) [upload=12231]: Done starting upload Aug 7 21:28:53.851804427 (11h49m15.595861277s) [upload=12231]: Uploading segments... Aug 7 21:28:53.851851609 (11h49m15.595908471s) [upload=12231]: Got next segment. Inline: true Aug 7 21:28:53.851857251 (11h49m15.595914111s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:28:53.851899816 (11h49m15.595956668s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:53.851905032 (11h49m15.595961892s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:53.851908856 (11h49m15.595965718s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:54.191384948 (11h49m15.935441808s) [upload=12231]: Done uploading segments... Aug 7 21:28:54.191405192 (11h49m15.935462045s) [upload=12231]: Upload finished. err: <nil> Aug 7 21:29:19.999054333 (11h49m41.74311118s) [upload=12237]: Starting upload Aug 7 21:29:19.999128859 (11h49m41.743185722s) [upload=12237]: Done starting upload Aug 7 21:29:19.999188080 (11h49m41.743244931s) [upload=12237]: Uploading segments... Aug 7 21:29:19.999327277 (11h49m41.74338415s) [upload=12237]: Got next segment. Inline: true Aug 7 21:29:19.999335397 (11h49m41.743392265s) [upload=12237]: Waiting for error group managing segments... Aug 7 21:29:19.999418844 (11h49m41.743475691s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:19.999426848 (11h49m41.743483712s) [upload=12237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:19.999430264 (11h49m41.743487128s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:21.165098561 (11h49m42.909155421s) [upload=12237]: Done uploading segments... Aug 7 21:29:21.165118654 (11h49m42.909175505s) [upload=12237]: Upload finished. err: <nil> Aug 7 21:29:59.841735510 (11h50m21.585792373s) [upload=12245]: Starting upload Aug 7 21:29:59.841811008 (11h50m21.585867874s) [upload=12245]: Done starting upload Aug 7 21:29:59.841889062 (11h50m21.585945909s) [upload=12245]: Uploading segments... Aug 7 21:29:59.841961319 (11h50m21.58601819s) [upload=12245]: Got next segment. Inline: true Aug 7 21:29:59.841966490 (11h50m21.586023336s) [upload=12245]: Waiting for error group managing segments... Aug 7 21:29:59.842009456 (11h50m21.586066303s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.842013678 (11h50m21.586070524s) [upload=12245]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.842016402 (11h50m21.586073249s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.937966407 (11h50m21.682023257s) [upload=12245]: Done uploading segments... Aug 7 21:29:59.937989326 (11h50m21.682046187s) [upload=12245]: Upload finished. err: metaclient: Too Many Requests Duration: 967.696265ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 19 out of 60 is failed (threshold=3) [........................................X.XXXXXXXXXXXXXXXXXX] duration: 1m0.000648243s FAIL (2025-08-07 21:34:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3s) [upload=12162]: Uploading segments... Aug 7 21:24:14.664378438 (11h44m36.408435308s) [upload=12162]: Got next segment. Inline: true Aug 7 21:24:14.664385109 (11h44m36.408441977s) [upload=12162]: Waiting for error group managing segments... Aug 7 21:24:14.664444486 (11h44m36.408501361s) [upload=12162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:14.664452219 (11h44m36.408509087s) [upload=12162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:14.664455983 (11h44m36.40851285s) [upload=12162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:14.390008254 (11h45m36.134065107s) [upload=12162]: Done uploading segments... Aug 7 21:25:14.390037098 (11h45m36.134093947s) [upload=12162]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:14.891337243 (11h45m36.635394094s) [upload=12178]: Starting upload Aug 7 21:25:14.891405973 (11h45m36.635462834s) [upload=12178]: Done starting upload Aug 7 21:25:14.891587260 (11h45m36.635644108s) [upload=12178]: Uploading segments... Aug 7 21:25:14.891659588 (11h45m36.635716456s) [upload=12178]: Got next segment. Inline: true Aug 7 21:25:14.891666181 (11h45m36.635723042s) [upload=12178]: Waiting for error group managing segments... Aug 7 21:25:14.891713596 (11h45m36.635770446s) [upload=12178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:14.891719665 (11h45m36.635776525s) [upload=12178]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:14.891723701 (11h45m36.635780562s) [upload=12178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:14.618805682 (11h46m36.362862552s) [upload=12178]: Done uploading segments... Aug 7 21:26:14.618848987 (11h46m36.36290586s) [upload=12178]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:15.042634192 (11h46m36.786691042s) [upload=12195]: Starting upload Aug 7 21:26:15.042707477 (11h46m36.786764348s) [upload=12195]: Done starting upload Aug 7 21:26:15.042793174 (11h46m36.786850024s) [upload=12195]: Uploading segments... Aug 7 21:26:15.042848386 (11h46m36.78690526s) [upload=12195]: Got next segment. Inline: true Aug 7 21:26:15.042855793 (11h46m36.786912664s) [upload=12195]: Waiting for error group managing segments... Aug 7 21:26:15.042911164 (11h46m36.786968025s) [upload=12195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:15.042918440 (11h46m36.786975312s) [upload=12195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:15.042924061 (11h46m36.78698093s) [upload=12195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:14.771724725 (11h47m36.515781576s) [upload=12195]: Done uploading segments... Aug 7 21:27:14.771758030 (11h47m36.515814885s) [upload=12195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:15.202352221 (11h47m36.946409081s) [upload=12209]: Starting upload Aug 7 21:27:15.202440356 (11h47m36.946497217s) [upload=12209]: Done starting upload Aug 7 21:27:15.202525888 (11h47m36.946582739s) [upload=12209]: Uploading segments... Aug 7 21:27:15.202588330 (11h47m36.946645196s) [upload=12209]: Got next segment. Inline: true Aug 7 21:27:15.202594295 (11h47m36.946651156s) [upload=12209]: Waiting for error group managing segments... Aug 7 21:27:15.202651093 (11h47m36.946707944s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:15.202656763 (11h47m36.946713624s) [upload=12209]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:15.202660199 (11h47m36.946717059s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:14.931851536 (11h48m36.675908396s) [upload=12209]: Done uploading segments... Aug 7 21:28:14.931874519 (11h48m36.67593138s) [upload=12209]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:21.072819157 (11h48m42.816876017s) [upload=12223]: Starting upload Aug 7 21:28:21.072899208 (11h48m42.816956068s) [upload=12223]: Done starting upload Aug 7 21:28:21.072981263 (11h48m42.817038113s) [upload=12223]: Uploading segments... Aug 7 21:28:21.073042235 (11h48m42.817099103s) [upload=12223]: Got next segment. Inline: true Aug 7 21:28:21.073047353 (11h48m42.817104214s) [upload=12223]: Waiting for error group managing segments... Aug 7 21:28:21.073104067 (11h48m42.817160927s) [upload=12223]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:21.073109154 (11h48m42.817166016s) [upload=12223]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:21.073112378 (11h48m42.817169238s) [upload=12223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:20.797200985 (11h49m42.541258063s) [upload=12223]: Done uploading segments... Aug 7 21:29:20.797233820 (11h49m42.541290687s) [upload=12223]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:21.230279976 (11h49m42.974336838s) [upload=12238]: Starting upload Aug 7 21:29:21.230391607 (11h49m42.974448467s) [upload=12238]: Done starting upload Aug 7 21:29:21.230482429 (11h49m42.974539278s) [upload=12238]: Uploading segments... Aug 7 21:29:21.230606006 (11h49m42.974662867s) [upload=12238]: Got next segment. Inline: true Aug 7 21:29:21.230613609 (11h49m42.974670469s) [upload=12238]: Waiting for error group managing segments... Aug 7 21:29:21.230660759 (11h49m42.974717609s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:21.230666510 (11h49m42.974723374s) [upload=12238]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:21.230691185 (11h49m42.974748046s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:20.959347392 (11h50m42.703404243s) [upload=12238]: Done uploading segments... Aug 7 21:30:20.959370210 (11h50m42.70342706s) [upload=12238]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:21.398814384 (11h50m43.142871245s) [upload=12254]: Starting upload Aug 7 21:30:21.398880649 (11h50m43.142937512s) [upload=12254]: Done starting upload Aug 7 21:30:21.398949225 (11h50m43.143006078s) [upload=12254]: Uploading segments... Aug 7 21:30:21.398993888 (11h50m43.143050757s) [upload=12254]: Got next segment. Inline: true Aug 7 21:30:21.399003181 (11h50m43.143060046s) [upload=12254]: Waiting for error group managing segments... Aug 7 21:30:21.399046484 (11h50m43.143103345s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:21.399052138 (11h50m43.143108999s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:21.399055744 (11h50m43.143112604s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:21.120945605 (11h51m42.865002833s) [upload=12254]: Done uploading segments... Aug 7 21:31:21.121002610 (11h51m42.865059457s) [upload=12254]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:21.580865540 (11h51m43.324922404s) [upload=12267]: Starting upload Aug 7 21:31:21.580962495 (11h51m43.32501936s) [upload=12267]: Done starting upload Aug 7 21:31:21.581042226 (11h51m43.325099087s) [upload=12267]: Uploading segments... Aug 7 21:31:21.581118360 (11h51m43.32517523s) [upload=12267]: Got next segment. Inline: true Aug 7 21:31:21.581124116 (11h51m43.325180981s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:31:21.581178813 (11h51m43.325235677s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:21.581184674 (11h51m43.325241538s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:21.581188187 (11h51m43.325245048s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:21.285242448 (11h52m43.029299295s) [upload=12267]: Done uploading segments... Aug 7 21:32:21.285264804 (11h52m43.029321651s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.849527306 (11h52m52.593584169s) [upload=12284]: Starting upload Aug 7 21:32:30.849593974 (11h52m52.593650837s) [upload=12284]: Done starting upload Aug 7 21:32:30.849656936 (11h52m52.593713781s) [upload=12284]: Uploading segments... Aug 7 21:32:30.849696837 (11h52m52.593753706s) [upload=12284]: Got next segment. Inline: true Aug 7 21:32:30.849701844 (11h52m52.593758708s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:32:30.849743987 (11h52m52.593800834s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.849748481 (11h52m52.59380534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.849753457 (11h52m52.593810316s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.580274752 (11h53m52.3243316s) [upload=12284]: Done uploading segments... Aug 7 21:33:30.580297199 (11h53m52.324354049s) [upload=12284]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:30.996524574 (11h53m52.740581425s) [upload=12301]: Starting upload Aug 7 21:33:30.996601123 (11h53m52.740657983s) [upload=12301]: Done starting upload Aug 7 21:33:30.996679475 (11h53m52.740736329s) [upload=12301]: Uploading segments... Aug 7 21:33:30.996753641 (11h53m52.740810668s) [upload=12301]: Got next segment. Inline: true Aug 7 21:33:30.996760213 (11h53m52.740817074s) [upload=12301]: Waiting for error group managing segments... Aug 7 21:33:30.996825729 (11h53m52.740882584s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:30.996832674 (11h53m52.740889525s) [upload=12301]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:30.996838492 (11h53m52.740895343s) [upload=12301]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:34:30.718542438 (11h54m52.462599299s) [upload=12301]: Done uploading segments... Aug 7 21:34:30.718564899 (11h54m52.462621759s) [upload=12301]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000664974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 28 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000696178s FAIL (2025-08-07 21:34:29 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000722043s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XX..............XX.] duration: 297.140063ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:34:12 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 297.211063ms edge gateway vhost us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XX..............XX] duration: 2m0.00101605s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:34:11 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.001030707s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000734796s FAIL (2025-08-07 21:34:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000761285s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..............................X..XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001275956s FAIL (2025-08-07 21:34:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002077542s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X......................X.......................XX.X] duration: 370.041837ms FAIL (2025-08-07 21:34:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== 979s) [noise=true , address=104.36.82.44:4892 , piece_id=W4NSH7UTQ34P5IZLD3XCALGBL6YZ5QJFVFFYVOF4DZYE4PLGC7EA, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.304266198 (11h54m5.048323067s) [noise=true , address=104.36.82.15:4261 , piece_id=ZN2CAPB5FV7XPEVJARIIFNPOPEY2GOEFQFRTZZNZLPLT765MRULA, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.304638399 (11h54m5.048695263s) [noise=true , address=104.36.82.4:4827 , piece_id=EGB2BTVBBNBDN57ZRD4WFTEYQ6GJCSUEKP3I6UI6XYEGFSWVQLWA, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.304964313 (11h54m5.049021182s) [noise=true , address=79.127.205.234:4585 , piece_id=JZO4TRJX7CFYUKI2U7QBY54FS3NRL2GG66FDTWAKHQIDQ6ZUWSFA, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.305289194 (11h54m5.049346062s) [noise=true , address=104.36.82.19:4405 , piece_id=F2TDP4KIXX7SX6JX2FOKOILDTEDBSBC2Y4XB4HO7J6NZOSSTMXCQ, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.305663222 (11h54m5.049720088s) [noise=true , address=104.36.82.59:4303 , piece_id=HLXP2T7IJOJSVQTD6X4UJWZZ4E3HC6AXRGRTN2OUP4P3YXL4HBUQ, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.307375967 (11h54m5.051432827s) [noise=true , address=64.49.13.140:4506 , piece_id=Y7SSPSS7GYN7ZVI3JFV2KRPSPWKWUS44M6BPZKJCYD5NPBKZC7LA, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.307994205 (11h54m5.052051068s) [noise=true , address=104.36.82.34:4578 , piece_id=WWMNXVJFSSU3AUT5DSWNAFTTZMNFK7COL6D6P2CEH4A4O73BRDJQ, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.308330311 (11h54m5.052387178s) [noise=true , address=104.36.82.45:4922 , piece_id=XKGSL3DS6JSY345IEK25AWBGSDX2IWUWDCDVXA66GBMPEO3WQWTQ, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.308696609 (11h54m5.052753472s) [noise=true , address=104.36.82.23:4462 , piece_id=CX6GMAOJFTZG2YUAQKRLCJYE2725ORXZL64IV34B2TIEPICFYXSQ, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.309027383 (11h54m5.053084243s) [noise=true , address=104.36.82.16:4297 , piece_id=Z5ACR62PJJNYADTFWU36W6L7GZF5NYSH7KT2J66UIY63NT3U3RHA, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.309517639 (11h54m5.053574502s) [noise=true , address=104.36.82.3:4770 , piece_id=2DHQP37D3NH2DVJTM7QANBATIVW4KP7XAXLPTJHEQN3GBQGV4F2Q, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.310005964 (11h54m5.054062831s) [noise=true , address=104.36.82.37:4666 , piece_id=ONYCF5TCS7H2NQNPK53UHS5HQORAZLVTCYNOFU7JAVREM5EQJSMA, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.310590144 (11h54m5.05464703s) [noise=true , address=104.36.82.60:4338 , piece_id=A6H2W5I3YT7NXCNCJTYZ4UFGIEDNZNRISA6RLHEMZOCUUPHYZYMQ, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.310999480 (11h54m5.055056348s) [noise=true , address=45.91.116.11:4132 , piece_id=YL7AGZAI2AMUKWML22F64Z65AKJZR2Z3WOEUDZBNOK4IWRTSS4DQ, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.311301215 (11h54m5.055358078s) [noise=true , address=79.127.205.233:4623 , piece_id=76LKINU73EN5LDGNSBLZRTLZCWHT45L25VUP4HWRPHO2ABV6CZDA, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.311594224 (11h54m5.055651092s) [noise=true , address=104.36.82.62:4417 , piece_id=JFFKIHB6YJZJLTQ5CLGATME7KA6B6TIML6XZCGYUQBFWRXAEB7DA, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.311902248 (11h54m5.055959119s) [noise=true , address=104.36.82.13:4189 , piece_id=RERQLVB7KU3V37Z6QVOJ5HGHV6IWXWOMJYPQMRBKLUM7ER3DUTCQ, seg_pos={0 0}, upload=12304]: Uploading piece... Aug 7 21:33:43.887542534 (11h54m5.631599386s) [upload=12304]: Next returned nil segment Aug 7 21:33:43.887567861 (11h54m5.631624821s) [upload=12304]: Waiting for error group managing segments... Aug 7 21:33:44.048562222 (11h54m5.792619083s) [noise=true , address=104.36.82.37:4666 , piece_id=ONYCF5TCS7H2NQNPK53UHS5HQORAZLVTCYNOFU7JAVREM5EQJSMA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil> Aug 7 21:33:44.059436730 (11h54m5.80349358s) [noise=true , address=104.36.82.19:4405 , piece_id=F2TDP4KIXX7SX6JX2FOKOILDTEDBSBC2Y4XB4HO7J6NZOSSTMXCQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil> Aug 7 21:33:44.067299265 (11h54m5.811356118s) [noise=true , address=104.36.82.60:4338 , piece_id=A6H2W5I3YT7NXCNCJTYZ4UFGIEDNZNRISA6RLHEMZOCUUPHYZYMQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil> Aug 7 21:33:44.113365621 (11h54m5.857422471s) [noise=true , address=104.36.82.13:4189 , piece_id=RERQLVB7KU3V37Z6QVOJ5HGHV6IWXWOMJYPQMRBKLUM7ER3DUTCQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil> Aug 7 21:33:44.113990013 (11h54m5.858046868s) [noise=true , address=104.36.82.15:4261 , piece_id=ZN2CAPB5FV7XPEVJARIIFNPOPEY2GOEFQFRTZZNZLPLT765MRULA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil> Aug 7 21:33:44.129055509 (11h54m5.873112362s) [noise=true , address=104.36.82.4:4827 , piece_id=EGB2BTVBBNBDN57ZRD4WFTEYQ6GJCSUEKP3I6UI6XYEGFSWVQLWA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil> Aug 7 21:33:44.174268649 (11h54m5.918325499s) [noise=true , address=45.91.116.11:4132 , piece_id=YL7AGZAI2AMUKWML22F64Z65AKJZR2Z3WOEUDZBNOK4IWRTSS4DQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil> Aug 7 21:33:44.212907217 (11h54m5.956964063s) [noise=true , address=104.36.82.59:4303 , piece_id=HLXP2T7IJOJSVQTD6X4UJWZZ4E3HC6AXRGRTN2OUP4P3YXL4HBUQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil> Aug 7 21:33:44.234310642 (11h54m5.978367495s) [noise=true , address=104.36.82.62:4417 , piece_id=JFFKIHB6YJZJLTQ5CLGATME7KA6B6TIML6XZCGYUQBFWRXAEB7DA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil> Aug 7 21:33:44.270648309 (11h54m6.014705171s) [noise=true , address=104.36.82.34:4578 , piece_id=WWMNXVJFSSU3AUT5DSWNAFTTZMNFK7COL6D6P2CEH4A4O73BRDJQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil> Aug 7 21:33:44.470354111 (11h54m6.214410971s) [noise=true , address=104.36.82.3:4770 , piece_id=2DHQP37D3NH2DVJTM7QANBATIVW4KP7XAXLPTJHEQN3GBQGV4F2Q, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil> Aug 7 21:33:44.489350040 (11h54m6.233406907s) [noise=true , address=104.36.82.44:4892 , piece_id=W4NSH7UTQ34P5IZLD3XCALGBL6YZ5QJFVFFYVOF4DZYE4PLGC7EA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil> Aug 7 21:33:44.624667783 (11h54m6.36872463s) [noise=true , address=104.36.82.45:4922 , piece_id=XKGSL3DS6JSY345IEK25AWBGSDX2IWUWDCDVXA66GBMPEO3WQWTQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil> Aug 7 21:33:44.641115000 (11h54m6.385171862s) [noise=true , address=79.127.205.233:4623 , piece_id=76LKINU73EN5LDGNSBLZRTLZCWHT45L25VUP4HWRPHO2ABV6CZDA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: <nil> Aug 7 21:33:44.641149015 (11h54m6.385205876s) [seg_pos={0 0}, upload=12304]: Segment reached optimal threshold of 14 pieces. Aug 7 21:33:44.641367964 (11h54m6.385424829s) [noise=true , address=64.49.13.140:4506 , piece_id=Y7SSPSS7GYN7ZVI3JFV2KRPSPWKWUS44M6BPZKJCYD5NPBKZC7LA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12LYkWHxg4uodPG98gJ66uz34YjsBN9N3iiy6rWQpnLjhiKoCoW): protocol: expected piece hash; context canceled; EOF Aug 7 21:33:44.641595473 (11h54m6.385652338s) [noise=true , address=79.127.205.234:4585 , piece_id=JZO4TRJX7CFYUKI2U7QBY54FS3NRL2GG66FDTWAKHQIDQ6ZUWSFA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GHc2AyFWF13heMfJDetaST47dXeeYuq1hB3ZjYwN7vUfXuvBF): protocol: expected piece hash; context canceled; EOF Aug 7 21:33:44.641716595 (11h54m6.385773458s) [noise=true , address=104.36.82.23:4462 , piece_id=CX6GMAOJFTZG2YUAQKRLCJYE2725ORXZL64IV34B2TIEPICFYXSQ, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YirC8WVSoE1qrWGMmayNmCsDLxjUhggrgXL5unKtQjsfUkbMu): protocol: expected piece hash; context canceled; EOF Aug 7 21:33:44.641795902 (11h54m6.385852754s) [noise=true , address=104.36.82.16:4297 , piece_id=Z5ACR62PJJNYADTFWU36W6L7GZF5NYSH7KT2J66UIY63NT3U3RHA, seg_pos={0 0}, upload=12304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g2pJxnoFW2TAzcAzNhjzDRJ3yYDK2mibG8tdX2Qy2YabNTq86): protocol: expected piece hash; context canceled; EOF Aug 7 21:33:44.641820947 (11h54m6.385877796s) [seg_pos={0 0}, upload=12304]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:33:44.641980472 (11h54m6.386037333s) [upload=12304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:33:44.641986313 (11h54m6.386043173s) [upload=12304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:44.811637442 (11h54m6.55569429s) [upload=12304]: Done uploading segments... Aug 7 21:33:44.811656829 (11h54m6.555713675s) [upload=12304]: Upload finished. err: <nil> Aug 7 21:34:01.932026219 (11h54m23.676083072s) [upload=12308]: Starting upload Aug 7 21:34:01.932128481 (11h54m23.676185355s) [upload=12308]: Done starting upload Aug 7 21:34:01.932434039 (11h54m23.676490888s) [upload=12308]: Uploading segments... Aug 7 21:34:01.932820553 (11h54m23.676877417s) [upload=12308]: Got next segment. Inline: false Aug 7 21:34:01.932925753 (11h54m23.676982622s) [upload=12308]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:34:01.932934357 (11h54m23.676991224s) [upload=12308]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:34:02.022671161 (11h54m23.766728061s) [upload=12308]: Next segment err: metaclient: Too Many Requests Aug 7 21:34:02.022701989 (11h54m23.766758862s) [upload=12308]: Done uploading segments... Aug 7 21:34:02.022759129 (11h54m23.766815993s) [upload=12308]: Upload finished. err: metaclient: Too Many Requests Duration: 370.084673ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 28 out of 60 is failed (threshold=3) [...............................XX.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001219396s FAIL (2025-08-07 21:33:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1.996758923s) [upload=12201]: Uploading segments... Aug 7 21:26:40.252782136 (11h47m1.996839003s) [upload=12201]: Got next segment. Inline: true Aug 7 21:26:40.252790092 (11h47m1.996846954s) [upload=12201]: Waiting for error group managing segments... Aug 7 21:26:40.252842463 (11h47m1.996899315s) [upload=12201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:40.252849740 (11h47m1.996906592s) [upload=12201]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:40.252854153 (11h47m1.996911015s) [upload=12201]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:09.981768433 (11h47m31.725825298s) [upload=12201]: Done uploading segments... Aug 7 21:27:09.981793631 (11h47m31.725850483s) [upload=12201]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:26.837217755 (11h47m48.581274619s) [upload=12211]: Starting upload Aug 7 21:27:26.837300223 (11h47m48.581357084s) [upload=12211]: Done starting upload Aug 7 21:27:26.837385984 (11h47m48.581442831s) [upload=12211]: Uploading segments... Aug 7 21:27:26.837444443 (11h47m48.581501313s) [upload=12211]: Got next segment. Inline: true Aug 7 21:27:26.837449770 (11h47m48.581506619s) [upload=12211]: Waiting for error group managing segments... Aug 7 21:27:26.837517257 (11h47m48.581574107s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:26.837523154 (11h47m48.581580003s) [upload=12211]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:26.837527097 (11h47m48.58158395s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:56.564982463 (11h48m18.309039313s) [upload=12211]: Done uploading segments... Aug 7 21:27:56.565004020 (11h48m18.309060871s) [upload=12211]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:36.367399060 (11h48m58.11145591s) [upload=12226]: Starting upload Aug 7 21:28:36.367588554 (11h48m58.111645417s) [upload=12226]: Done starting upload Aug 7 21:28:36.367664912 (11h48m58.111721765s) [upload=12226]: Uploading segments... Aug 7 21:28:36.367723895 (11h48m58.111780757s) [upload=12226]: Got next segment. Inline: true Aug 7 21:28:36.367729331 (11h48m58.111786192s) [upload=12226]: Waiting for error group managing segments... Aug 7 21:28:36.367785894 (11h48m58.111842754s) [upload=12226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:36.367791010 (11h48m58.11184787s) [upload=12226]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:36.367794492 (11h48m58.111851352s) [upload=12226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:06.092890267 (11h49m27.836947129s) [upload=12226]: Done uploading segments... Aug 7 21:29:06.092932205 (11h49m27.836989051s) [upload=12226]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:06.522310183 (11h49m28.266367033s) [upload=12233]: Starting upload Aug 7 21:29:06.522388772 (11h49m28.266445633s) [upload=12233]: Done starting upload Aug 7 21:29:06.522479144 (11h49m28.266535995s) [upload=12233]: Uploading segments... Aug 7 21:29:06.522560400 (11h49m28.266617269s) [upload=12233]: Got next segment. Inline: true Aug 7 21:29:06.522566995 (11h49m28.266623844s) [upload=12233]: Waiting for error group managing segments... Aug 7 21:29:06.522615251 (11h49m28.266672101s) [upload=12233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:06.522621757 (11h49m28.266678608s) [upload=12233]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:06.522630028 (11h49m28.266686879s) [upload=12233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:36.250046349 (11h49m57.994103211s) [upload=12233]: Done uploading segments... Aug 7 21:29:36.250088941 (11h49m57.994145808s) [upload=12233]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:03.947527959 (11h50m25.691584822s) [upload=12249]: Starting upload Aug 7 21:30:03.947767439 (11h50m25.691824311s) [upload=12249]: Done starting upload Aug 7 21:30:03.959515214 (11h50m25.703572066s) [upload=12249]: Uploading segments... Aug 7 21:30:03.959659046 (11h50m25.703715962s) [upload=12249]: Got next segment. Inline: true Aug 7 21:30:03.959668426 (11h50m25.703725302s) [upload=12249]: Waiting for error group managing segments... Aug 7 21:30:03.959783210 (11h50m25.703840082s) [upload=12249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:03.959790677 (11h50m25.703847531s) [upload=12249]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:03.959795320 (11h50m25.703852169s) [upload=12249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:33.655073335 (11h50m55.399130195s) [upload=12249]: Done uploading segments... Aug 7 21:30:33.655096560 (11h50m55.399153412s) [upload=12249]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:38.130475962 (11h50m59.874532829s) [upload=12256]: Starting upload Aug 7 21:30:38.130573915 (11h50m59.874630787s) [upload=12256]: Done starting upload Aug 7 21:30:38.130665890 (11h50m59.874722739s) [upload=12256]: Uploading segments... Aug 7 21:30:38.131017420 (11h50m59.875074302s) [upload=12256]: Got next segment. Inline: true Aug 7 21:30:38.131029217 (11h50m59.87508609s) [upload=12256]: Waiting for error group managing segments... Aug 7 21:30:38.131118811 (11h50m59.875175683s) [upload=12256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:38.131125270 (11h50m59.875182135s) [upload=12256]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:38.131131701 (11h50m59.875188569s) [upload=12256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:07.857077892 (11h51m29.601134744s) [upload=12256]: Done uploading segments... Aug 7 21:31:07.857100740 (11h51m29.601157585s) [upload=12256]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:11.159434284 (11h51m32.903491145s) [upload=12264]: Starting upload Aug 7 21:31:11.159626488 (11h51m32.903683349s) [upload=12264]: Done starting upload Aug 7 21:31:11.159711096 (11h51m32.903767956s) [upload=12264]: Uploading segments... Aug 7 21:31:11.159755148 (11h51m32.903812017s) [upload=12264]: Got next segment. Inline: true Aug 7 21:31:11.159760622 (11h51m32.903817485s) [upload=12264]: Waiting for error group managing segments... Aug 7 21:31:11.159814091 (11h51m32.903870951s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:11.159819842 (11h51m32.903876703s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:11.159823678 (11h51m32.903880539s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:40.887250927 (11h52m2.631307774s) [upload=12264]: Done uploading segments... Aug 7 21:31:40.887302452 (11h52m2.631359314s) [upload=12264]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:45.153828960 (11h52m6.89788582s) [upload=12272]: Starting upload Aug 7 21:31:45.153940931 (11h52m6.897997802s) [upload=12272]: Done starting upload Aug 7 21:31:45.154020734 (11h52m6.898077597s) [upload=12272]: Uploading segments... Aug 7 21:31:45.154082030 (11h52m6.898138908s) [upload=12272]: Got next segment. Inline: true Aug 7 21:31:45.154088359 (11h52m6.898145223s) [upload=12272]: Waiting for error group managing segments... Aug 7 21:31:45.154142898 (11h52m6.898199759s) [upload=12272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:45.154150494 (11h52m6.898207362s) [upload=12272]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:45.154155049 (11h52m6.898211914s) [upload=12272]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:14.861389353 (11h52m36.605446232s) [upload=12272]: Done uploading segments... Aug 7 21:32:14.861441235 (11h52m36.605498102s) [upload=12272]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:34.523168733 (11h52m56.267225597s) [upload=12286]: Starting upload Aug 7 21:32:34.523264231 (11h52m56.267321089s) [upload=12286]: Done starting upload Aug 7 21:32:34.523343267 (11h52m56.267400122s) [upload=12286]: Uploading segments... Aug 7 21:32:34.523393734 (11h52m56.267450607s) [upload=12286]: Got next segment. Inline: true Aug 7 21:32:34.523400482 (11h52m56.267457354s) [upload=12286]: Waiting for error group managing segments... Aug 7 21:32:34.523450495 (11h52m56.267507356s) [upload=12286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:34.523455899 (11h52m56.267512759s) [upload=12286]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:34.523555922 (11h52m56.267612773s) [upload=12286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:04.247355762 (11h53m25.991412634s) [upload=12286]: Done uploading segments... Aug 7 21:33:04.247398685 (11h53m25.991455557s) [upload=12286]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:33:27.128759856 (11h53m48.872816706s) [upload=12299]: Starting upload Aug 7 21:33:27.128845560 (11h53m48.872902422s) [upload=12299]: Done starting upload Aug 7 21:33:27.128936415 (11h53m48.872993265s) [upload=12299]: Uploading segments... Aug 7 21:33:27.128981290 (11h53m48.873038156s) [upload=12299]: Got next segment. Inline: true Aug 7 21:33:27.128986811 (11h53m48.873043661s) [upload=12299]: Waiting for error group managing segments... Aug 7 21:33:27.129031120 (11h53m48.87308798s) [upload=12299]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:27.129036464 (11h53m48.873093314s) [upload=12299]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:33:27.129041631 (11h53m48.873098481s) [upload=12299]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:56.852338652 (11h54m18.596395503s) [upload=12299]: Done uploading segments... Aug 7 21:33:56.852369847 (11h54m18.596426695s) [upload=12299]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001242683s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 30.00088934s FAIL (2025-08-07 21:33:56 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000915319s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...............................................XXXXXXXXXXXXX] duration: 30.000810137s FAIL (2025-08-07 21:33:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== EDOJGJR7CY6FW6PALYZMQRA7KNPSQQB4IAYHPQZRA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798662439 (11h40m35.542719308s) [noise=true , address=77.90.31.13:4627 , piece_id=TOKP44CMWL4OXYLWOFHL7WFKSN2C57WGHWBPZMQDVZ4GRDKLCBRA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798759384 (11h40m35.54281625s) [noise=true , address=77.90.31.10:4519 , piece_id=FDV33ZMCVYJOMT4YP7KIRA3MBB7OUNJHAPITYYLMXJW6DZWKGGCQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798971326 (11h40m35.54302818s) [noise=true , address=45.91.116.12:4225 , piece_id=EHBNQPDYM7HB4OGRJGU6QZ3CTM5YWHUEV2SSD5LKUZI6LGG6X4GA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.799201622 (11h40m35.543258483s) [noise=true , address=104.36.82.2:4731 , piece_id=DKWMN2CLFT3XVYTGEF3HJYZXPME5TE37I45TD5ABWVRQMWE3X6YA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.799847282 (11h40m35.543904151s) [noise=true , address=77.90.31.13:4626 , piece_id=VNGCTMK7ET2RCVPEDCDXZHYKARZIIGUU6CPVT6LKA7HCLWJFJ7AQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.800536275 (11h40m35.544593146s) [noise=true , address=45.91.116.5:4297 , piece_id=77LOHOMR6FRLTTYGTAVC4SHMSDSDWGNITJMIIKDIA7ADS2272TAQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.800663553 (11h40m35.544720423s) [noise=true , address=45.91.116.12:4226 , piece_id=47PAAFR7VRN7H7MBS7Z7KRONLGN6Q2JF5CBYT6ASUIANYBUSYNKQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:14.404668625 (11h40m36.148725475s) [upload=12096]: Next returned nil segment Aug 7 21:20:14.404685815 (11h40m36.148742676s) [upload=12096]: Waiting for error group managing segments... Aug 7 21:20:14.561430060 (11h40m36.305486911s) [noise=true , address=45.91.116.12:4225 , piece_id=EHBNQPDYM7HB4OGRJGU6QZ3CTM5YWHUEV2SSD5LKUZI6LGG6X4GA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.583124069 (11h40m36.327180919s) [noise=true , address=104.36.82.15:4257 , piece_id=46HGSNBRDPF3JZMTPWY3OPAGK3RXBGTZWXDDNWZQERV6QAYWJ3TA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.622563656 (11h40m36.366620519s) [noise=true , address=45.91.116.8:4047 , piece_id=RCBP5UBGZ5C6YEQQQ3SD3GUU26LJTFX3GOL4XZOKEFZOXB4WDRSA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.648162660 (11h40m36.392219508s) [noise=true , address=77.90.31.17:4772 , piece_id=EMV6OLIN7YSBQQN4SC4VJGWQGFKLPBQVVJZTTKUE5ELQDXMXPSSQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.649278924 (11h40m36.393335781s) [noise=true , address=104.36.82.7:4912 , piece_id=2XMUVIQB6QCEDOJGJR7CY6FW6PALYZMQRA7KNPSQQB4IAYHPQZRA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.657067139 (11h40m36.401123988s) [noise=true , address=77.90.31.18:4807 , piece_id=O3SDUB6BQB6CPYN4X7Z7GHUK5Y663IOBGE4S4MB5UBB46HCQMJLA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.660665476 (11h40m36.404722337s) [noise=true , address=77.90.31.13:4627 , piece_id=TOKP44CMWL4OXYLWOFHL7WFKSN2C57WGHWBPZMQDVZ4GRDKLCBRA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.692922160 (11h40m36.43697902s) [noise=true , address=104.36.82.2:4731 , piece_id=DKWMN2CLFT3XVYTGEF3HJYZXPME5TE37I45TD5ABWVRQMWE3X6YA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.693482368 (11h40m36.437539243s) [noise=true , address=45.91.116.12:4226 , piece_id=47PAAFR7VRN7H7MBS7Z7KRONLGN6Q2JF5CBYT6ASUIANYBUSYNKQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.722004664 (11h40m36.466061524s) [noise=true , address=45.91.116.5:4297 , piece_id=77LOHOMR6FRLTTYGTAVC4SHMSDSDWGNITJMIIKDIA7ADS2272TAQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.747287536 (11h40m36.491344386s) [noise=true , address=45.91.116.6:4973 , piece_id=UKM3AZMNW67WQ7WMFP5Z55NG4JBZBFLDVODUHIISQYZ27HUVHP6Q, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.751701687 (11h40m36.495758555s) [noise=true , address=77.90.31.18:4805 , piece_id=HKKJRNDDMITSXEXDDMZ37ZCG5OFNG72PAEDN7TNWDHXZQYBSWIXA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.798490539 (11h40m36.542547386s) [noise=true , address=77.90.31.10:4519 , piece_id=FDV33ZMCVYJOMT4YP7KIRA3MBB7OUNJHAPITYYLMXJW6DZWKGGCQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.798534450 (11h40m36.54259131s) [seg_pos={0 0}, upload=12096]: Segment reached optimal threshold of 13 pieces. Aug 7 21:20:14.798673827 (11h40m36.542730687s) [noise=true , address=45.91.116.10:4082 , piece_id=3AL7IMQKZU26B34QQEK43SAXSWT6W7M4TF4ITTYF774U2LED2LRQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798750267 (11h40m36.542807123s) [noise=true , address=77.90.31.13:4626 , piece_id=VNGCTMK7ET2RCVPEDCDXZHYKARZIIGUU6CPVT6LKA7HCLWJFJ7AQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798798579 (11h40m36.542855426s) [noise=true , address=45.91.116.4:4937 , piece_id=PGZP47Q6KAXPGOPJ5JAAKADDBRWMAIBX5GWL6JGHFJUU3AMOPKTQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798820360 (11h40m36.54287721s) [seg_pos={0 0}, upload=12096]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:20:14.798910105 (11h40m36.542966966s) [upload=12096]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:20:14.798915686 (11h40m36.542972547s) [upload=12096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:24.464709964 (11h51m46.208766834s) [upload=12096]: Done uploading segments... Aug 7 21:31:24.464745783 (11h51m46.208802653s) [upload=12096]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:24.881904902 (11h51m46.625961766s) [upload=12268]: Starting upload Aug 7 21:31:24.882094151 (11h51m46.626151016s) [upload=12268]: Done starting upload Aug 7 21:31:24.882164223 (11h51m46.626221082s) [upload=12268]: Uploading segments... Aug 7 21:31:24.882475408 (11h51m46.626532277s) [upload=12268]: Got next segment. Inline: false Aug 7 21:31:24.882552261 (11h51m46.626609127s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:24.882560165 (11h51m46.626617032s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:54.614102730 (11h52m16.358159588s) [upload=12268]: Next segment err: context deadline exceeded Aug 7 21:31:54.614205863 (11h52m16.358262714s) [upload=12268]: Done uploading segments... Aug 7 21:31:54.614225924 (11h52m16.358282773s) [upload=12268]: Upload finished. err: context deadline exceeded Aug 7 21:32:00.253568814 (11h52m21.997625667s) [upload=12278]: Starting upload Aug 7 21:32:00.253669214 (11h52m21.997726066s) [upload=12278]: Done starting upload Aug 7 21:32:00.254008956 (11h52m21.998065808s) [upload=12278]: Uploading segments... Aug 7 21:32:00.254068254 (11h52m21.998125125s) [upload=12278]: Got next segment. Inline: false Aug 7 21:32:00.254112779 (11h52m21.998169639s) [upload=12278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:00.254118157 (11h52m21.998175018s) [upload=12278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:29.984781139 (11h52m51.728838008s) [upload=12278]: Next segment err: context deadline exceeded Aug 7 21:32:29.984827795 (11h52m51.728884645s) [upload=12278]: Done uploading segments... Aug 7 21:32:29.984851910 (11h52m51.728908763s) [upload=12278]: Upload finished. err: context deadline exceeded Aug 7 21:32:35.321237619 (11h52m57.065294469s) [upload=12287]: Starting upload Aug 7 21:32:35.321310476 (11h52m57.065367326s) [upload=12287]: Done starting upload Aug 7 21:32:35.321418401 (11h52m57.06547525s) [upload=12287]: Uploading segments... Aug 7 21:32:35.321610690 (11h52m57.06566754s) [upload=12287]: Got next segment. Inline: false Aug 7 21:32:35.321654358 (11h52m57.065711209s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:35.321659987 (11h52m57.065716847s) [upload=12287]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:05.046284716 (11h53m26.790341587s) [upload=12287]: Next segment err: context deadline exceeded Aug 7 21:33:05.046403894 (11h53m26.790460755s) [upload=12287]: Done uploading segments... Aug 7 21:33:05.046426465 (11h53m26.790483318s) [upload=12287]: Upload finished. err: context deadline exceeded Aug 7 21:33:21.055953150 (11h53m42.80001s) [upload=12296]: Starting upload Aug 7 21:33:21.056022172 (11h53m42.800079036s) [upload=12296]: Done starting upload Aug 7 21:33:21.056266284 (11h53m42.800323139s) [upload=12296]: Uploading segments... Aug 7 21:33:21.056360175 (11h53m42.800417035s) [upload=12296]: Got next segment. Inline: false Aug 7 21:33:21.056419681 (11h53m42.800476544s) [upload=12296]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:33:21.056426657 (11h53m42.800483517s) [upload=12296]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:50.766946667 (11h54m12.511003536s) [upload=12296]: Next segment err: context deadline exceeded Aug 7 21:33:50.767114018 (11h54m12.511170869s) [upload=12296]: Done uploading segments... Aug 7 21:33:50.767133351 (11h54m12.511190205s) [upload=12296]: Upload finished. err: context deadline exceeded Duration: 30.00082825s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X......................X.......................XX.] duration: 3.037328059s FAIL (2025-08-07 21:33:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 3858f464b8e19c011a6473aa117db5b2ddf588631921b96f5bbe513c02bc1ab7 != before d6eafb9b08554dd09d4923938b27af1cb088f21ad32a3334d034e6c9d5e328c6 ======LOGS====== 7 21:33:24.318168236 (11h53m46.062225082s) [seg_pos={0 0}, upload=12297]: Done begin upload segment. Aug 7 21:33:24.318192727 (11h53m46.06224959s) [noise=true , address=104.36.82.12:4166 , piece_id=FS7AE5MRLMZ4GSC4FQOWD3RGUHL67UV7PT2ZA3TLTJ2KWT7VNPRA, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.318209704 (11h53m46.062266574s) [noise=true , address=104.36.82.34:4565 , piece_id=JRRPV7GMYCWXOCXJT6JBUMIGITB5EKAK4726MYEMFTVS5FEOQRUA, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.318540984 (11h53m46.062597831s) [noise=true , address=104.36.82.23:4480 , piece_id=CPMCLYPVEXUKWX4E7N4HQGXD7S2ZDP67W7UE2OA7YVMTQEVRQOBQ, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.318721301 (11h53m46.062778164s) [noise=true , address=104.36.82.9:4021 , piece_id=ARNCZUEDPD45VC2ODEWLXQGVZMFYSLABST7VFXYMAVX7R32ZAPGQ, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.318791810 (11h53m46.062848656s) [noise=true , address=104.36.82.2:4764 , piece_id=XNGVIPGQQ63WH4I3G42EP3QABDKQPRKI7W2Q2BTPDV5KMI47LYFA, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.319042294 (11h53m46.063099176s) [noise=true , address=104.36.82.44:4901 , piece_id=3TUJK3TSAHVXUJINCKDNW2XKX2MDPKU56ZPAEYCIXBTV7D32NANA, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.319115506 (11h53m46.063172368s) [noise=true , address=64.49.13.170:4726 , piece_id=2UYNF4J6IWOUCBZH2OVSGFUAQRRH4IDYHVD6DIGJS3I66BMHOE4Q, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.319624205 (11h53m46.063681075s) [noise=true , address=104.36.82.55:4228 , piece_id=RJXJAPAPTU5F5S37XNEPU6HTFIL4KE2EXWFIBPOCCODSNVBUURWA, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.319715747 (11h53m46.063772613s) [noise=true , address=104.36.82.54:4169 , piece_id=6WLXLSXUGH5G4CKLCOA5776JCIYQX7KNVICDGNGSXMUWYILNU3IQ, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.319986583 (11h53m46.064043433s) [noise=true , address=104.36.82.62:4429 , piece_id=QRGVQKKNW4OEB7ICEWHMJGGQCESD3KU2DPNPXWT7V6SBRHQWQN6Q, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.320105587 (11h53m46.064162458s) [noise=true , address=45.91.116.12:4251 , piece_id=SDUV3M6STWF6EXRAB3ZIYTKUBY45HOM7VVTCGI5XKVROJKNQAGHA, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.320301196 (11h53m46.064358066s) [noise=true , address=104.36.82.36:4638 , piece_id=QMZCQ5OSQVHFXXVHWQA736KIVUOYTZRNLNQUT6PHID7YLZVA6N2Q, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.320571435 (11h53m46.064628298s) [noise=true , address=104.36.82.59:4335 , piece_id=3WYDBY7Z4X3QDVK4GH2IP7MXHORFTB53R5X2W7MSJ7NTHS7RWLJQ, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.320619401 (11h53m46.064676252s) [noise=true , address=79.127.201.215:4351 , piece_id=IBFCK4YAF2SJ6UP65MCNCLJNCP6IJDMQXZ2M5RYEHSZM64WE7K6Q, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.320936029 (11h53m46.064992886s) [noise=true , address=104.36.82.60:4363 , piece_id=FPKTD3VKIANIK5LYMXUTXW5LKM6TPA3C2J73TRKJ42BOZOPQXORA, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.321346741 (11h53m46.065403589s) [noise=true , address=104.36.82.46:4492 , piece_id=UJKXA4UAYXWTX4DSNFYLCGJNQCXF4OJD7GTYVWPTGZL3X7GJ6FPQ, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.322200959 (11h53m46.066257811s) [noise=true , address=104.36.82.61:4381 , piece_id=RDLZEAEZEP7CH4PZXFUBOAAJXOGLYTBL276ZIMHMSSZPPNZHUVYQ, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.322543795 (11h53m46.066600645s) [noise=true , address=104.36.82.45:4932 , piece_id=DFAP5DIB64W5CAX4B3BKYWG227MSOWATO2KYK3XEJ7B76GLWHY4A, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.904809696 (11h53m46.648866558s) [upload=12297]: Next returned nil segment Aug 7 21:33:24.904828986 (11h53m46.648885851s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:33:25.089044086 (11h53m46.833100939s) [noise=true , address=104.36.82.9:4021 , piece_id=ARNCZUEDPD45VC2ODEWLXQGVZMFYSLABST7VFXYMAVX7R32ZAPGQ, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.091410992 (11h53m46.835467851s) [noise=true , address=104.36.82.59:4335 , piece_id=3WYDBY7Z4X3QDVK4GH2IP7MXHORFTB53R5X2W7MSJ7NTHS7RWLJQ, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.112317040 (11h53m46.856373888s) [noise=true , address=104.36.82.34:4565 , piece_id=JRRPV7GMYCWXOCXJT6JBUMIGITB5EKAK4726MYEMFTVS5FEOQRUA, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.150930998 (11h53m46.894987862s) [noise=true , address=104.36.82.55:4228 , piece_id=RJXJAPAPTU5F5S37XNEPU6HTFIL4KE2EXWFIBPOCCODSNVBUURWA, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.207983268 (11h53m46.952040121s) [noise=true , address=104.36.82.54:4169 , piece_id=6WLXLSXUGH5G4CKLCOA5776JCIYQX7KNVICDGNGSXMUWYILNU3IQ, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.217723374 (11h53m46.96178023s) [noise=true , address=104.36.82.12:4166 , piece_id=FS7AE5MRLMZ4GSC4FQOWD3RGUHL67UV7PT2ZA3TLTJ2KWT7VNPRA, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.260663362 (11h53m47.004720215s) [noise=true , address=45.91.116.12:4251 , piece_id=SDUV3M6STWF6EXRAB3ZIYTKUBY45HOM7VVTCGI5XKVROJKNQAGHA, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.351819073 (11h53m47.095875923s) [noise=true , address=104.36.82.61:4381 , piece_id=RDLZEAEZEP7CH4PZXFUBOAAJXOGLYTBL276ZIMHMSSZPPNZHUVYQ, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.395435586 (11h53m47.139492434s) [noise=true , address=79.127.201.215:4351 , piece_id=IBFCK4YAF2SJ6UP65MCNCLJNCP6IJDMQXZ2M5RYEHSZM64WE7K6Q, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.455443947 (11h53m47.1995008s) [noise=true , address=104.36.82.36:4638 , piece_id=QMZCQ5OSQVHFXXVHWQA736KIVUOYTZRNLNQUT6PHID7YLZVA6N2Q, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.569354068 (11h53m47.313410915s) [noise=true , address=64.49.13.170:4726 , piece_id=2UYNF4J6IWOUCBZH2OVSGFUAQRRH4IDYHVD6DIGJS3I66BMHOE4Q, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.582723736 (11h53m47.326780598s) [noise=true , address=104.36.82.23:4480 , piece_id=CPMCLYPVEXUKWX4E7N4HQGXD7S2ZDP67W7UE2OA7YVMTQEVRQOBQ, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.632550869 (11h53m47.376607719s) [noise=true , address=104.36.82.46:4492 , piece_id=UJKXA4UAYXWTX4DSNFYLCGJNQCXF4OJD7GTYVWPTGZL3X7GJ6FPQ, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:26.021685966 (11h53m47.765742819s) [noise=true , address=104.36.82.62:4429 , piece_id=QRGVQKKNW4OEB7ICEWHMJGGQCESD3KU2DPNPXWT7V6SBRHQWQN6Q, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:26.021706465 (11h53m47.765763316s) [seg_pos={0 0}, upload=12297]: Segment reached optimal threshold of 14 pieces. Aug 7 21:33:26.021847094 (11h53m47.765903945s) [noise=true , address=104.36.82.45:4932 , piece_id=DFAP5DIB64W5CAX4B3BKYWG227MSOWATO2KYK3XEJ7B76GLWHY4A, seg_pos={0 0}, upload=12297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF Aug 7 21:33:26.022119083 (11h53m47.766175929s) [noise=true , address=104.36.82.2:4764 , piece_id=XNGVIPGQQ63WH4I3G42EP3QABDKQPRKI7W2Q2BTPDV5KMI47LYFA, seg_pos={0 0}, upload=12297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hC8kZEorpoPUHZE9fKmpYEEG6HrSWPkn574p4LehpoGwPHPdt): protocol: write tcp 139.59.144.22:39320->104.36.82.2:4764: use of closed network connection; write tcp 139.59.144.22:39320->104.36.82.2:4764: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39320->104.36.82.2:4764: use of closed network connection Aug 7 21:33:26.022171833 (11h53m47.766228693s) [noise=true , address=104.36.82.60:4363 , piece_id=FPKTD3VKIANIK5LYMXUTXW5LKM6TPA3C2J73TRKJ42BOZOPQXORA, seg_pos={0 0}, upload=12297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: write tcp 139.59.144.22:37214->104.36.82.60:4363: use of closed network connection; write tcp 139.59.144.22:37214->104.36.82.60:4363: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37214->104.36.82.60:4363: use of closed network connection Aug 7 21:33:26.022240372 (11h53m47.766297237s) [noise=true , address=104.36.82.44:4901 , piece_id=3TUJK3TSAHVXUJINCKDNW2XKX2MDPKU56ZPAEYCIXBTV7D32NANA, seg_pos={0 0}, upload=12297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 139.59.144.22:48376->104.36.82.44:4901: use of closed network connection; write tcp 139.59.144.22:48376->104.36.82.44:4901: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48376->104.36.82.44:4901: use of closed network connection Aug 7 21:33:26.022296474 (11h53m47.766353323s) [seg_pos={0 0}, upload=12297]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:33:26.022385335 (11h53m47.766442186s) [upload=12297]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:33:26.022390402 (11h53m47.766447253s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:26.247932089 (11h53m47.991988959s) [upload=12297]: Done uploading segments... Aug 7 21:33:26.247974371 (11h53m47.992031235s) [upload=12297]: Upload finished. err: <nil> Duration: 3.037370907s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 886.897628ms FAIL (2025-08-07 21:33:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== loading segments... Aug 7 21:21:33.493714811 (11h41m55.237771657s) [upload=12125]: Upload finished. err: <nil> Aug 7 21:22:47.998800022 (11h43m9.742856869s) [upload=12141]: Starting upload Aug 7 21:22:47.998849424 (11h43m9.742906283s) [upload=12141]: Done starting upload Aug 7 21:22:47.998919383 (11h43m9.742976233s) [upload=12141]: Uploading segments... Aug 7 21:22:47.999001069 (11h43m9.743057938s) [upload=12141]: Got next segment. Inline: true Aug 7 21:22:47.999007866 (11h43m9.743064718s) [upload=12141]: Waiting for error group managing segments... Aug 7 21:22:47.999066684 (11h43m9.743123533s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:47.999073371 (11h43m9.743130221s) [upload=12141]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:47.999077096 (11h43m9.743133947s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:48.164602559 (11h43m9.90865943s) [upload=12141]: Done uploading segments... Aug 7 21:22:48.164642998 (11h43m9.908699853s) [upload=12141]: Upload finished. err: <nil> Aug 7 21:23:35.990065388 (11h43m57.734122239s) [upload=12154]: Starting upload Aug 7 21:23:35.990143300 (11h43m57.734200146s) [upload=12154]: Done starting upload Aug 7 21:23:35.990217262 (11h43m57.734274108s) [upload=12154]: Uploading segments... Aug 7 21:23:35.990277036 (11h43m57.734333925s) [upload=12154]: Got next segment. Inline: true Aug 7 21:23:35.990283708 (11h43m57.734340569s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:23:35.990337388 (11h43m57.734394253s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:35.990343849 (11h43m57.734400714s) [upload=12154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:35.990347914 (11h43m57.734405001s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:36.150463692 (11h43m57.894520544s) [upload=12154]: Done uploading segments... Aug 7 21:23:36.150483857 (11h43m57.894540707s) [upload=12154]: Upload finished. err: <nil> Aug 7 21:24:13.830997960 (11h44m35.57505482s) [upload=12160]: Starting upload Aug 7 21:24:13.831049433 (11h44m35.575106295s) [upload=12160]: Done starting upload Aug 7 21:24:13.831151824 (11h44m35.575208678s) [upload=12160]: Uploading segments... Aug 7 21:24:13.831192510 (11h44m35.575249377s) [upload=12160]: Got next segment. Inline: true Aug 7 21:24:13.831197614 (11h44m35.575254475s) [upload=12160]: Waiting for error group managing segments... Aug 7 21:24:13.831253497 (11h44m35.57531036s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:13.831259191 (11h44m35.575316046s) [upload=12160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:13.831262688 (11h44m35.575319551s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:13.982951222 (11h44m35.727008092s) [upload=12160]: Done uploading segments... Aug 7 21:24:13.982975878 (11h44m35.727032727s) [upload=12160]: Upload finished. err: <nil> Aug 7 21:24:48.198960455 (11h45m9.943017328s) [upload=12172]: Starting upload Aug 7 21:24:48.199050501 (11h45m9.943107348s) [upload=12172]: Done starting upload Aug 7 21:24:48.199166478 (11h45m9.943223341s) [upload=12172]: Uploading segments... Aug 7 21:24:48.199284289 (11h45m9.943341159s) [upload=12172]: Got next segment. Inline: true Aug 7 21:24:48.199309162 (11h45m9.943366036s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:24:48.199360469 (11h45m9.943417336s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:48.199382302 (11h45m9.943439163s) [upload=12172]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:48.199386864 (11h45m9.943443734s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:48.353366598 (11h45m10.09742347s) [upload=12172]: Done uploading segments... Aug 7 21:24:48.353410465 (11h45m10.097467327s) [upload=12172]: Upload finished. err: <nil> Aug 7 21:25:56.601363668 (11h46m18.345420518s) [upload=12190]: Starting upload Aug 7 21:25:56.601416202 (11h46m18.345473063s) [upload=12190]: Done starting upload Aug 7 21:25:56.601478987 (11h46m18.345535837s) [upload=12190]: Uploading segments... Aug 7 21:25:56.601519567 (11h46m18.345576436s) [upload=12190]: Got next segment. Inline: true Aug 7 21:25:56.601524613 (11h46m18.345581463s) [upload=12190]: Waiting for error group managing segments... Aug 7 21:25:56.601564530 (11h46m18.345621381s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:56.601571562 (11h46m18.345628412s) [upload=12190]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:56.601574952 (11h46m18.345631806s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.289489478 (11h46m19.033546339s) [upload=12190]: Done uploading segments... Aug 7 21:25:57.289514011 (11h46m19.033570861s) [upload=12190]: Upload finished. err: <nil> Aug 7 21:26:32.814119549 (11h46m54.558176399s) [upload=12198]: Starting upload Aug 7 21:26:32.814178314 (11h46m54.558235164s) [upload=12198]: Done starting upload Aug 7 21:26:32.814230962 (11h46m54.558287814s) [upload=12198]: Uploading segments... Aug 7 21:26:32.814314990 (11h46m54.558371858s) [upload=12198]: Got next segment. Inline: true Aug 7 21:26:32.814321493 (11h46m54.558378353s) [upload=12198]: Waiting for error group managing segments... Aug 7 21:26:32.814387101 (11h46m54.558443951s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.814393685 (11h46m54.558450545s) [upload=12198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:32.814397318 (11h46m54.558454179s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:32.972796167 (11h46m54.716853027s) [upload=12198]: Done uploading segments... Aug 7 21:26:32.972815783 (11h46m54.716872632s) [upload=12198]: Upload finished. err: <nil> Aug 7 21:27:41.280232963 (11h48m3.024289826s) [upload=12214]: Starting upload Aug 7 21:27:41.280291948 (11h48m3.024348811s) [upload=12214]: Done starting upload Aug 7 21:27:41.280410716 (11h48m3.024467565s) [upload=12214]: Uploading segments... Aug 7 21:27:41.280478626 (11h48m3.024535487s) [upload=12214]: Got next segment. Inline: true Aug 7 21:27:41.280485326 (11h48m3.024542193s) [upload=12214]: Waiting for error group managing segments... Aug 7 21:27:41.280533304 (11h48m3.024590156s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:41.280538628 (11h48m3.024595488s) [upload=12214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:41.280542274 (11h48m3.02459914s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:41.433978896 (11h48m3.178035773s) [upload=12214]: Done uploading segments... Aug 7 21:27:41.434003733 (11h48m3.178060594s) [upload=12214]: Upload finished. err: <nil> Aug 7 21:28:53.851663561 (11h49m15.595720412s) [upload=12231]: Starting upload Aug 7 21:28:53.851736366 (11h49m15.595793226s) [upload=12231]: Done starting upload Aug 7 21:28:53.851804427 (11h49m15.595861277s) [upload=12231]: Uploading segments... Aug 7 21:28:53.851851609 (11h49m15.595908471s) [upload=12231]: Got next segment. Inline: true Aug 7 21:28:53.851857251 (11h49m15.595914111s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:28:53.851899816 (11h49m15.595956668s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:53.851905032 (11h49m15.595961892s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:53.851908856 (11h49m15.595965718s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:54.191384948 (11h49m15.935441808s) [upload=12231]: Done uploading segments... Aug 7 21:28:54.191405192 (11h49m15.935462045s) [upload=12231]: Upload finished. err: <nil> Aug 7 21:29:19.999054333 (11h49m41.74311118s) [upload=12237]: Starting upload Aug 7 21:29:19.999128859 (11h49m41.743185722s) [upload=12237]: Done starting upload Aug 7 21:29:19.999188080 (11h49m41.743244931s) [upload=12237]: Uploading segments... Aug 7 21:29:19.999327277 (11h49m41.74338415s) [upload=12237]: Got next segment. Inline: true Aug 7 21:29:19.999335397 (11h49m41.743392265s) [upload=12237]: Waiting for error group managing segments... Aug 7 21:29:19.999418844 (11h49m41.743475691s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:19.999426848 (11h49m41.743483712s) [upload=12237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:19.999430264 (11h49m41.743487128s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:21.165098561 (11h49m42.909155421s) [upload=12237]: Done uploading segments... Aug 7 21:29:21.165118654 (11h49m42.909175505s) [upload=12237]: Upload finished. err: <nil> Aug 7 21:29:59.841735510 (11h50m21.585792373s) [upload=12245]: Starting upload Aug 7 21:29:59.841811008 (11h50m21.585867874s) [upload=12245]: Done starting upload Aug 7 21:29:59.841889062 (11h50m21.585945909s) [upload=12245]: Uploading segments... Aug 7 21:29:59.841961319 (11h50m21.58601819s) [upload=12245]: Got next segment. Inline: true Aug 7 21:29:59.841966490 (11h50m21.586023336s) [upload=12245]: Waiting for error group managing segments... Aug 7 21:29:59.842009456 (11h50m21.586066303s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.842013678 (11h50m21.586070524s) [upload=12245]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.842016402 (11h50m21.586073249s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.937966407 (11h50m21.682023257s) [upload=12245]: Done uploading segments... Aug 7 21:29:59.937989326 (11h50m21.682046187s) [upload=12245]: Upload finished. err: metaclient: Too Many Requests Duration: 886.929795ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X............X.] duration: 111.913672ms remote: "tcp:185.244.226.2:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 21:33:42 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 111.928898ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 1m0.001266649s FAIL (2025-08-07 21:33:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001282744s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001391535s FAIL (2025-08-07 21:33:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001422284s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 27 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001686727s FAIL (2025-08-07 21:33:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001717007s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [...............................X..XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001075964s FAIL (2025-08-07 21:33:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00110617s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 18 out of 60 is failed (threshold=3) [.........................................X.XXXXXXXXXXXXXXXXX] duration: 1m0.001400437s FAIL (2025-08-07 21:33:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 3s) [upload=12148]: Uploading segments... Aug 7 21:23:14.506132768 (11h43m36.250189638s) [upload=12148]: Got next segment. Inline: true Aug 7 21:23:14.506140538 (11h43m36.250197403s) [upload=12148]: Waiting for error group managing segments... Aug 7 21:23:14.506211738 (11h43m36.250268603s) [upload=12148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:14.506218667 (11h43m36.250275531s) [upload=12148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:14.506222979 (11h43m36.250279842s) [upload=12148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:14.230441064 (11h44m35.974497931s) [upload=12148]: Done uploading segments... Aug 7 21:24:14.230468679 (11h44m35.974525551s) [upload=12148]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:14.664138591 (11h44m36.408195463s) [upload=12162]: Starting upload Aug 7 21:24:14.664235374 (11h44m36.408292237s) [upload=12162]: Done starting upload Aug 7 21:24:14.664324426 (11h44m36.408381273s) [upload=12162]: Uploading segments... Aug 7 21:24:14.664378438 (11h44m36.408435308s) [upload=12162]: Got next segment. Inline: true Aug 7 21:24:14.664385109 (11h44m36.408441977s) [upload=12162]: Waiting for error group managing segments... Aug 7 21:24:14.664444486 (11h44m36.408501361s) [upload=12162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:14.664452219 (11h44m36.408509087s) [upload=12162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:14.664455983 (11h44m36.40851285s) [upload=12162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:14.390008254 (11h45m36.134065107s) [upload=12162]: Done uploading segments... Aug 7 21:25:14.390037098 (11h45m36.134093947s) [upload=12162]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:14.891337243 (11h45m36.635394094s) [upload=12178]: Starting upload Aug 7 21:25:14.891405973 (11h45m36.635462834s) [upload=12178]: Done starting upload Aug 7 21:25:14.891587260 (11h45m36.635644108s) [upload=12178]: Uploading segments... Aug 7 21:25:14.891659588 (11h45m36.635716456s) [upload=12178]: Got next segment. Inline: true Aug 7 21:25:14.891666181 (11h45m36.635723042s) [upload=12178]: Waiting for error group managing segments... Aug 7 21:25:14.891713596 (11h45m36.635770446s) [upload=12178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:14.891719665 (11h45m36.635776525s) [upload=12178]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:14.891723701 (11h45m36.635780562s) [upload=12178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:14.618805682 (11h46m36.362862552s) [upload=12178]: Done uploading segments... Aug 7 21:26:14.618848987 (11h46m36.36290586s) [upload=12178]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:15.042634192 (11h46m36.786691042s) [upload=12195]: Starting upload Aug 7 21:26:15.042707477 (11h46m36.786764348s) [upload=12195]: Done starting upload Aug 7 21:26:15.042793174 (11h46m36.786850024s) [upload=12195]: Uploading segments... Aug 7 21:26:15.042848386 (11h46m36.78690526s) [upload=12195]: Got next segment. Inline: true Aug 7 21:26:15.042855793 (11h46m36.786912664s) [upload=12195]: Waiting for error group managing segments... Aug 7 21:26:15.042911164 (11h46m36.786968025s) [upload=12195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:15.042918440 (11h46m36.786975312s) [upload=12195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:15.042924061 (11h46m36.78698093s) [upload=12195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:14.771724725 (11h47m36.515781576s) [upload=12195]: Done uploading segments... Aug 7 21:27:14.771758030 (11h47m36.515814885s) [upload=12195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:15.202352221 (11h47m36.946409081s) [upload=12209]: Starting upload Aug 7 21:27:15.202440356 (11h47m36.946497217s) [upload=12209]: Done starting upload Aug 7 21:27:15.202525888 (11h47m36.946582739s) [upload=12209]: Uploading segments... Aug 7 21:27:15.202588330 (11h47m36.946645196s) [upload=12209]: Got next segment. Inline: true Aug 7 21:27:15.202594295 (11h47m36.946651156s) [upload=12209]: Waiting for error group managing segments... Aug 7 21:27:15.202651093 (11h47m36.946707944s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:15.202656763 (11h47m36.946713624s) [upload=12209]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:15.202660199 (11h47m36.946717059s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:14.931851536 (11h48m36.675908396s) [upload=12209]: Done uploading segments... Aug 7 21:28:14.931874519 (11h48m36.67593138s) [upload=12209]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:21.072819157 (11h48m42.816876017s) [upload=12223]: Starting upload Aug 7 21:28:21.072899208 (11h48m42.816956068s) [upload=12223]: Done starting upload Aug 7 21:28:21.072981263 (11h48m42.817038113s) [upload=12223]: Uploading segments... Aug 7 21:28:21.073042235 (11h48m42.817099103s) [upload=12223]: Got next segment. Inline: true Aug 7 21:28:21.073047353 (11h48m42.817104214s) [upload=12223]: Waiting for error group managing segments... Aug 7 21:28:21.073104067 (11h48m42.817160927s) [upload=12223]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:21.073109154 (11h48m42.817166016s) [upload=12223]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:21.073112378 (11h48m42.817169238s) [upload=12223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:20.797200985 (11h49m42.541258063s) [upload=12223]: Done uploading segments... Aug 7 21:29:20.797233820 (11h49m42.541290687s) [upload=12223]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:21.230279976 (11h49m42.974336838s) [upload=12238]: Starting upload Aug 7 21:29:21.230391607 (11h49m42.974448467s) [upload=12238]: Done starting upload Aug 7 21:29:21.230482429 (11h49m42.974539278s) [upload=12238]: Uploading segments... Aug 7 21:29:21.230606006 (11h49m42.974662867s) [upload=12238]: Got next segment. Inline: true Aug 7 21:29:21.230613609 (11h49m42.974670469s) [upload=12238]: Waiting for error group managing segments... Aug 7 21:29:21.230660759 (11h49m42.974717609s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:21.230666510 (11h49m42.974723374s) [upload=12238]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:21.230691185 (11h49m42.974748046s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:20.959347392 (11h50m42.703404243s) [upload=12238]: Done uploading segments... Aug 7 21:30:20.959370210 (11h50m42.70342706s) [upload=12238]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:21.398814384 (11h50m43.142871245s) [upload=12254]: Starting upload Aug 7 21:30:21.398880649 (11h50m43.142937512s) [upload=12254]: Done starting upload Aug 7 21:30:21.398949225 (11h50m43.143006078s) [upload=12254]: Uploading segments... Aug 7 21:30:21.398993888 (11h50m43.143050757s) [upload=12254]: Got next segment. Inline: true Aug 7 21:30:21.399003181 (11h50m43.143060046s) [upload=12254]: Waiting for error group managing segments... Aug 7 21:30:21.399046484 (11h50m43.143103345s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:21.399052138 (11h50m43.143108999s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:21.399055744 (11h50m43.143112604s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:21.120945605 (11h51m42.865002833s) [upload=12254]: Done uploading segments... Aug 7 21:31:21.121002610 (11h51m42.865059457s) [upload=12254]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:21.580865540 (11h51m43.324922404s) [upload=12267]: Starting upload Aug 7 21:31:21.580962495 (11h51m43.32501936s) [upload=12267]: Done starting upload Aug 7 21:31:21.581042226 (11h51m43.325099087s) [upload=12267]: Uploading segments... Aug 7 21:31:21.581118360 (11h51m43.32517523s) [upload=12267]: Got next segment. Inline: true Aug 7 21:31:21.581124116 (11h51m43.325180981s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:31:21.581178813 (11h51m43.325235677s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:21.581184674 (11h51m43.325241538s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:21.581188187 (11h51m43.325245048s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:21.285242448 (11h52m43.029299295s) [upload=12267]: Done uploading segments... Aug 7 21:32:21.285264804 (11h52m43.029321651s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:30.849527306 (11h52m52.593584169s) [upload=12284]: Starting upload Aug 7 21:32:30.849593974 (11h52m52.593650837s) [upload=12284]: Done starting upload Aug 7 21:32:30.849656936 (11h52m52.593713781s) [upload=12284]: Uploading segments... Aug 7 21:32:30.849696837 (11h52m52.593753706s) [upload=12284]: Got next segment. Inline: true Aug 7 21:32:30.849701844 (11h52m52.593758708s) [upload=12284]: Waiting for error group managing segments... Aug 7 21:32:30.849743987 (11h52m52.593800834s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:30.849748481 (11h52m52.59380534s) [upload=12284]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:30.849753457 (11h52m52.593810316s) [upload=12284]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:30.580274752 (11h53m52.3243316s) [upload=12284]: Done uploading segments... Aug 7 21:33:30.580297199 (11h53m52.324354049s) [upload=12284]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.00142606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X......................X.......................XX] duration: 4.129404839s FAIL (2025-08-07 21:33:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 3858f464b8e19c011a6473aa117db5b2ddf588631921b96f5bbe513c02bc1ab7 != before d6eafb9b08554dd09d4923938b27af1cb088f21ad32a3334d034e6c9d5e328c6 ======LOGS====== 7 21:33:24.318168236 (11h53m46.062225082s) [seg_pos={0 0}, upload=12297]: Done begin upload segment. Aug 7 21:33:24.318192727 (11h53m46.06224959s) [noise=true , address=104.36.82.12:4166 , piece_id=FS7AE5MRLMZ4GSC4FQOWD3RGUHL67UV7PT2ZA3TLTJ2KWT7VNPRA, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.318209704 (11h53m46.062266574s) [noise=true , address=104.36.82.34:4565 , piece_id=JRRPV7GMYCWXOCXJT6JBUMIGITB5EKAK4726MYEMFTVS5FEOQRUA, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.318540984 (11h53m46.062597831s) [noise=true , address=104.36.82.23:4480 , piece_id=CPMCLYPVEXUKWX4E7N4HQGXD7S2ZDP67W7UE2OA7YVMTQEVRQOBQ, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.318721301 (11h53m46.062778164s) [noise=true , address=104.36.82.9:4021 , piece_id=ARNCZUEDPD45VC2ODEWLXQGVZMFYSLABST7VFXYMAVX7R32ZAPGQ, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.318791810 (11h53m46.062848656s) [noise=true , address=104.36.82.2:4764 , piece_id=XNGVIPGQQ63WH4I3G42EP3QABDKQPRKI7W2Q2BTPDV5KMI47LYFA, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.319042294 (11h53m46.063099176s) [noise=true , address=104.36.82.44:4901 , piece_id=3TUJK3TSAHVXUJINCKDNW2XKX2MDPKU56ZPAEYCIXBTV7D32NANA, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.319115506 (11h53m46.063172368s) [noise=true , address=64.49.13.170:4726 , piece_id=2UYNF4J6IWOUCBZH2OVSGFUAQRRH4IDYHVD6DIGJS3I66BMHOE4Q, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.319624205 (11h53m46.063681075s) [noise=true , address=104.36.82.55:4228 , piece_id=RJXJAPAPTU5F5S37XNEPU6HTFIL4KE2EXWFIBPOCCODSNVBUURWA, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.319715747 (11h53m46.063772613s) [noise=true , address=104.36.82.54:4169 , piece_id=6WLXLSXUGH5G4CKLCOA5776JCIYQX7KNVICDGNGSXMUWYILNU3IQ, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.319986583 (11h53m46.064043433s) [noise=true , address=104.36.82.62:4429 , piece_id=QRGVQKKNW4OEB7ICEWHMJGGQCESD3KU2DPNPXWT7V6SBRHQWQN6Q, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.320105587 (11h53m46.064162458s) [noise=true , address=45.91.116.12:4251 , piece_id=SDUV3M6STWF6EXRAB3ZIYTKUBY45HOM7VVTCGI5XKVROJKNQAGHA, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.320301196 (11h53m46.064358066s) [noise=true , address=104.36.82.36:4638 , piece_id=QMZCQ5OSQVHFXXVHWQA736KIVUOYTZRNLNQUT6PHID7YLZVA6N2Q, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.320571435 (11h53m46.064628298s) [noise=true , address=104.36.82.59:4335 , piece_id=3WYDBY7Z4X3QDVK4GH2IP7MXHORFTB53R5X2W7MSJ7NTHS7RWLJQ, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.320619401 (11h53m46.064676252s) [noise=true , address=79.127.201.215:4351 , piece_id=IBFCK4YAF2SJ6UP65MCNCLJNCP6IJDMQXZ2M5RYEHSZM64WE7K6Q, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.320936029 (11h53m46.064992886s) [noise=true , address=104.36.82.60:4363 , piece_id=FPKTD3VKIANIK5LYMXUTXW5LKM6TPA3C2J73TRKJ42BOZOPQXORA, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.321346741 (11h53m46.065403589s) [noise=true , address=104.36.82.46:4492 , piece_id=UJKXA4UAYXWTX4DSNFYLCGJNQCXF4OJD7GTYVWPTGZL3X7GJ6FPQ, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.322200959 (11h53m46.066257811s) [noise=true , address=104.36.82.61:4381 , piece_id=RDLZEAEZEP7CH4PZXFUBOAAJXOGLYTBL276ZIMHMSSZPPNZHUVYQ, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.322543795 (11h53m46.066600645s) [noise=true , address=104.36.82.45:4932 , piece_id=DFAP5DIB64W5CAX4B3BKYWG227MSOWATO2KYK3XEJ7B76GLWHY4A, seg_pos={0 0}, upload=12297]: Uploading piece... Aug 7 21:33:24.904809696 (11h53m46.648866558s) [upload=12297]: Next returned nil segment Aug 7 21:33:24.904828986 (11h53m46.648885851s) [upload=12297]: Waiting for error group managing segments... Aug 7 21:33:25.089044086 (11h53m46.833100939s) [noise=true , address=104.36.82.9:4021 , piece_id=ARNCZUEDPD45VC2ODEWLXQGVZMFYSLABST7VFXYMAVX7R32ZAPGQ, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.091410992 (11h53m46.835467851s) [noise=true , address=104.36.82.59:4335 , piece_id=3WYDBY7Z4X3QDVK4GH2IP7MXHORFTB53R5X2W7MSJ7NTHS7RWLJQ, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.112317040 (11h53m46.856373888s) [noise=true , address=104.36.82.34:4565 , piece_id=JRRPV7GMYCWXOCXJT6JBUMIGITB5EKAK4726MYEMFTVS5FEOQRUA, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.150930998 (11h53m46.894987862s) [noise=true , address=104.36.82.55:4228 , piece_id=RJXJAPAPTU5F5S37XNEPU6HTFIL4KE2EXWFIBPOCCODSNVBUURWA, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.207983268 (11h53m46.952040121s) [noise=true , address=104.36.82.54:4169 , piece_id=6WLXLSXUGH5G4CKLCOA5776JCIYQX7KNVICDGNGSXMUWYILNU3IQ, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.217723374 (11h53m46.96178023s) [noise=true , address=104.36.82.12:4166 , piece_id=FS7AE5MRLMZ4GSC4FQOWD3RGUHL67UV7PT2ZA3TLTJ2KWT7VNPRA, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.260663362 (11h53m47.004720215s) [noise=true , address=45.91.116.12:4251 , piece_id=SDUV3M6STWF6EXRAB3ZIYTKUBY45HOM7VVTCGI5XKVROJKNQAGHA, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.351819073 (11h53m47.095875923s) [noise=true , address=104.36.82.61:4381 , piece_id=RDLZEAEZEP7CH4PZXFUBOAAJXOGLYTBL276ZIMHMSSZPPNZHUVYQ, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.395435586 (11h53m47.139492434s) [noise=true , address=79.127.201.215:4351 , piece_id=IBFCK4YAF2SJ6UP65MCNCLJNCP6IJDMQXZ2M5RYEHSZM64WE7K6Q, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.455443947 (11h53m47.1995008s) [noise=true , address=104.36.82.36:4638 , piece_id=QMZCQ5OSQVHFXXVHWQA736KIVUOYTZRNLNQUT6PHID7YLZVA6N2Q, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.569354068 (11h53m47.313410915s) [noise=true , address=64.49.13.170:4726 , piece_id=2UYNF4J6IWOUCBZH2OVSGFUAQRRH4IDYHVD6DIGJS3I66BMHOE4Q, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.582723736 (11h53m47.326780598s) [noise=true , address=104.36.82.23:4480 , piece_id=CPMCLYPVEXUKWX4E7N4HQGXD7S2ZDP67W7UE2OA7YVMTQEVRQOBQ, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:25.632550869 (11h53m47.376607719s) [noise=true , address=104.36.82.46:4492 , piece_id=UJKXA4UAYXWTX4DSNFYLCGJNQCXF4OJD7GTYVWPTGZL3X7GJ6FPQ, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:26.021685966 (11h53m47.765742819s) [noise=true , address=104.36.82.62:4429 , piece_id=QRGVQKKNW4OEB7ICEWHMJGGQCESD3KU2DPNPXWT7V6SBRHQWQN6Q, seg_pos={0 0}, upload=12297]: Done uploading piece. err: <nil> Aug 7 21:33:26.021706465 (11h53m47.765763316s) [seg_pos={0 0}, upload=12297]: Segment reached optimal threshold of 14 pieces. Aug 7 21:33:26.021847094 (11h53m47.765903945s) [noise=true , address=104.36.82.45:4932 , piece_id=DFAP5DIB64W5CAX4B3BKYWG227MSOWATO2KYK3XEJ7B76GLWHY4A, seg_pos={0 0}, upload=12297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF Aug 7 21:33:26.022119083 (11h53m47.766175929s) [noise=true , address=104.36.82.2:4764 , piece_id=XNGVIPGQQ63WH4I3G42EP3QABDKQPRKI7W2Q2BTPDV5KMI47LYFA, seg_pos={0 0}, upload=12297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hC8kZEorpoPUHZE9fKmpYEEG6HrSWPkn574p4LehpoGwPHPdt): protocol: write tcp 139.59.144.22:39320->104.36.82.2:4764: use of closed network connection; write tcp 139.59.144.22:39320->104.36.82.2:4764: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:39320->104.36.82.2:4764: use of closed network connection Aug 7 21:33:26.022171833 (11h53m47.766228693s) [noise=true , address=104.36.82.60:4363 , piece_id=FPKTD3VKIANIK5LYMXUTXW5LKM6TPA3C2J73TRKJ42BOZOPQXORA, seg_pos={0 0}, upload=12297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1endke3P5J92vFytHPiVczH5VcuHDSXyvd4qK9kFNhkQh8MhL7): protocol: write tcp 139.59.144.22:37214->104.36.82.60:4363: use of closed network connection; write tcp 139.59.144.22:37214->104.36.82.60:4363: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37214->104.36.82.60:4363: use of closed network connection Aug 7 21:33:26.022240372 (11h53m47.766297237s) [noise=true , address=104.36.82.44:4901 , piece_id=3TUJK3TSAHVXUJINCKDNW2XKX2MDPKU56ZPAEYCIXBTV7D32NANA, seg_pos={0 0}, upload=12297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1): protocol: write tcp 139.59.144.22:48376->104.36.82.44:4901: use of closed network connection; write tcp 139.59.144.22:48376->104.36.82.44:4901: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48376->104.36.82.44:4901: use of closed network connection Aug 7 21:33:26.022296474 (11h53m47.766353323s) [seg_pos={0 0}, upload=12297]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:33:26.022385335 (11h53m47.766442186s) [upload=12297]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:33:26.022390402 (11h53m47.766447253s) [upload=12297]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:26.247932089 (11h53m47.991988959s) [upload=12297]: Done uploading segments... Aug 7 21:33:26.247974371 (11h53m47.992031235s) [upload=12297]: Upload finished. err: <nil> Duration: 4.129437309s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 20 out of 60 is failed (threshold=3) [........................................XXXXXXXXXXXXXXXXXXXX] duration: 30.001592917s FAIL (2025-08-07 21:33:17 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.001610125s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 23 out of 60 is failed (threshold=3) [.....................................XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001013945s FAIL (2025-08-07 21:33:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.0010383s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................................XXXXXXXXXXXX] duration: 30.001016068s FAIL (2025-08-07 21:33:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== [noise=true , address=45.91.116.4:4937 , piece_id=PGZP47Q6KAXPGOPJ5JAAKADDBRWMAIBX5GWL6JGHFJUU3AMOPKTQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.797841016 (11h40m35.541897882s) [noise=true , address=45.91.116.10:4082 , piece_id=3AL7IMQKZU26B34QQEK43SAXSWT6W7M4TF4ITTYF774U2LED2LRQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.797890836 (11h40m35.541947696s) [noise=true , address=45.91.116.6:4973 , piece_id=UKM3AZMNW67WQ7WMFP5Z55NG4JBZBFLDVODUHIISQYZ27HUVHP6Q, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798097154 (11h40m35.542154017s) [noise=true , address=45.91.116.8:4047 , piece_id=RCBP5UBGZ5C6YEQQQ3SD3GUU26LJTFX3GOL4XZOKEFZOXB4WDRSA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798331623 (11h40m35.542388488s) [noise=true , address=104.36.82.7:4912 , piece_id=2XMUVIQB6QCEDOJGJR7CY6FW6PALYZMQRA7KNPSQQB4IAYHPQZRA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798662439 (11h40m35.542719308s) [noise=true , address=77.90.31.13:4627 , piece_id=TOKP44CMWL4OXYLWOFHL7WFKSN2C57WGHWBPZMQDVZ4GRDKLCBRA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798759384 (11h40m35.54281625s) [noise=true , address=77.90.31.10:4519 , piece_id=FDV33ZMCVYJOMT4YP7KIRA3MBB7OUNJHAPITYYLMXJW6DZWKGGCQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798971326 (11h40m35.54302818s) [noise=true , address=45.91.116.12:4225 , piece_id=EHBNQPDYM7HB4OGRJGU6QZ3CTM5YWHUEV2SSD5LKUZI6LGG6X4GA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.799201622 (11h40m35.543258483s) [noise=true , address=104.36.82.2:4731 , piece_id=DKWMN2CLFT3XVYTGEF3HJYZXPME5TE37I45TD5ABWVRQMWE3X6YA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.799847282 (11h40m35.543904151s) [noise=true , address=77.90.31.13:4626 , piece_id=VNGCTMK7ET2RCVPEDCDXZHYKARZIIGUU6CPVT6LKA7HCLWJFJ7AQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.800536275 (11h40m35.544593146s) [noise=true , address=45.91.116.5:4297 , piece_id=77LOHOMR6FRLTTYGTAVC4SHMSDSDWGNITJMIIKDIA7ADS2272TAQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.800663553 (11h40m35.544720423s) [noise=true , address=45.91.116.12:4226 , piece_id=47PAAFR7VRN7H7MBS7Z7KRONLGN6Q2JF5CBYT6ASUIANYBUSYNKQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:14.404668625 (11h40m36.148725475s) [upload=12096]: Next returned nil segment Aug 7 21:20:14.404685815 (11h40m36.148742676s) [upload=12096]: Waiting for error group managing segments... Aug 7 21:20:14.561430060 (11h40m36.305486911s) [noise=true , address=45.91.116.12:4225 , piece_id=EHBNQPDYM7HB4OGRJGU6QZ3CTM5YWHUEV2SSD5LKUZI6LGG6X4GA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.583124069 (11h40m36.327180919s) [noise=true , address=104.36.82.15:4257 , piece_id=46HGSNBRDPF3JZMTPWY3OPAGK3RXBGTZWXDDNWZQERV6QAYWJ3TA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.622563656 (11h40m36.366620519s) [noise=true , address=45.91.116.8:4047 , piece_id=RCBP5UBGZ5C6YEQQQ3SD3GUU26LJTFX3GOL4XZOKEFZOXB4WDRSA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.648162660 (11h40m36.392219508s) [noise=true , address=77.90.31.17:4772 , piece_id=EMV6OLIN7YSBQQN4SC4VJGWQGFKLPBQVVJZTTKUE5ELQDXMXPSSQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.649278924 (11h40m36.393335781s) [noise=true , address=104.36.82.7:4912 , piece_id=2XMUVIQB6QCEDOJGJR7CY6FW6PALYZMQRA7KNPSQQB4IAYHPQZRA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.657067139 (11h40m36.401123988s) [noise=true , address=77.90.31.18:4807 , piece_id=O3SDUB6BQB6CPYN4X7Z7GHUK5Y663IOBGE4S4MB5UBB46HCQMJLA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.660665476 (11h40m36.404722337s) [noise=true , address=77.90.31.13:4627 , piece_id=TOKP44CMWL4OXYLWOFHL7WFKSN2C57WGHWBPZMQDVZ4GRDKLCBRA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.692922160 (11h40m36.43697902s) [noise=true , address=104.36.82.2:4731 , piece_id=DKWMN2CLFT3XVYTGEF3HJYZXPME5TE37I45TD5ABWVRQMWE3X6YA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.693482368 (11h40m36.437539243s) [noise=true , address=45.91.116.12:4226 , piece_id=47PAAFR7VRN7H7MBS7Z7KRONLGN6Q2JF5CBYT6ASUIANYBUSYNKQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.722004664 (11h40m36.466061524s) [noise=true , address=45.91.116.5:4297 , piece_id=77LOHOMR6FRLTTYGTAVC4SHMSDSDWGNITJMIIKDIA7ADS2272TAQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.747287536 (11h40m36.491344386s) [noise=true , address=45.91.116.6:4973 , piece_id=UKM3AZMNW67WQ7WMFP5Z55NG4JBZBFLDVODUHIISQYZ27HUVHP6Q, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.751701687 (11h40m36.495758555s) [noise=true , address=77.90.31.18:4805 , piece_id=HKKJRNDDMITSXEXDDMZ37ZCG5OFNG72PAEDN7TNWDHXZQYBSWIXA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.798490539 (11h40m36.542547386s) [noise=true , address=77.90.31.10:4519 , piece_id=FDV33ZMCVYJOMT4YP7KIRA3MBB7OUNJHAPITYYLMXJW6DZWKGGCQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.798534450 (11h40m36.54259131s) [seg_pos={0 0}, upload=12096]: Segment reached optimal threshold of 13 pieces. Aug 7 21:20:14.798673827 (11h40m36.542730687s) [noise=true , address=45.91.116.10:4082 , piece_id=3AL7IMQKZU26B34QQEK43SAXSWT6W7M4TF4ITTYF774U2LED2LRQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798750267 (11h40m36.542807123s) [noise=true , address=77.90.31.13:4626 , piece_id=VNGCTMK7ET2RCVPEDCDXZHYKARZIIGUU6CPVT6LKA7HCLWJFJ7AQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798798579 (11h40m36.542855426s) [noise=true , address=45.91.116.4:4937 , piece_id=PGZP47Q6KAXPGOPJ5JAAKADDBRWMAIBX5GWL6JGHFJUU3AMOPKTQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798820360 (11h40m36.54287721s) [seg_pos={0 0}, upload=12096]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:20:14.798910105 (11h40m36.542966966s) [upload=12096]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:20:14.798915686 (11h40m36.542972547s) [upload=12096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:24.464709964 (11h51m46.208766834s) [upload=12096]: Done uploading segments... Aug 7 21:31:24.464745783 (11h51m46.208802653s) [upload=12096]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:24.881904902 (11h51m46.625961766s) [upload=12268]: Starting upload Aug 7 21:31:24.882094151 (11h51m46.626151016s) [upload=12268]: Done starting upload Aug 7 21:31:24.882164223 (11h51m46.626221082s) [upload=12268]: Uploading segments... Aug 7 21:31:24.882475408 (11h51m46.626532277s) [upload=12268]: Got next segment. Inline: false Aug 7 21:31:24.882552261 (11h51m46.626609127s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:24.882560165 (11h51m46.626617032s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:54.614102730 (11h52m16.358159588s) [upload=12268]: Next segment err: context deadline exceeded Aug 7 21:31:54.614205863 (11h52m16.358262714s) [upload=12268]: Done uploading segments... Aug 7 21:31:54.614225924 (11h52m16.358282773s) [upload=12268]: Upload finished. err: context deadline exceeded Aug 7 21:32:00.253568814 (11h52m21.997625667s) [upload=12278]: Starting upload Aug 7 21:32:00.253669214 (11h52m21.997726066s) [upload=12278]: Done starting upload Aug 7 21:32:00.254008956 (11h52m21.998065808s) [upload=12278]: Uploading segments... Aug 7 21:32:00.254068254 (11h52m21.998125125s) [upload=12278]: Got next segment. Inline: false Aug 7 21:32:00.254112779 (11h52m21.998169639s) [upload=12278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:00.254118157 (11h52m21.998175018s) [upload=12278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:29.984781139 (11h52m51.728838008s) [upload=12278]: Next segment err: context deadline exceeded Aug 7 21:32:29.984827795 (11h52m51.728884645s) [upload=12278]: Done uploading segments... Aug 7 21:32:29.984851910 (11h52m51.728908763s) [upload=12278]: Upload finished. err: context deadline exceeded Aug 7 21:32:35.321237619 (11h52m57.065294469s) [upload=12287]: Starting upload Aug 7 21:32:35.321310476 (11h52m57.065367326s) [upload=12287]: Done starting upload Aug 7 21:32:35.321418401 (11h52m57.06547525s) [upload=12287]: Uploading segments... Aug 7 21:32:35.321610690 (11h52m57.06566754s) [upload=12287]: Got next segment. Inline: false Aug 7 21:32:35.321654358 (11h52m57.065711209s) [upload=12287]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:35.321659987 (11h52m57.065716847s) [upload=12287]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:33:05.046284716 (11h53m26.790341587s) [upload=12287]: Next segment err: context deadline exceeded Aug 7 21:33:05.046403894 (11h53m26.790460755s) [upload=12287]: Done uploading segments... Aug 7 21:33:05.046426465 (11h53m26.790483318s) [upload=12287]: Upload finished. err: context deadline exceeded Duration: 30.001044846s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 27 out of 60 is failed (threshold=3) [................................XX.XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000816827s FAIL (2025-08-07 21:33:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== (11h46m7.633499729s) [upload=12186]: Uploading segments... Aug 7 21:25:45.889501484 (11h46m7.633558347s) [upload=12186]: Got next segment. Inline: true Aug 7 21:25:45.889507484 (11h46m7.633564344s) [upload=12186]: Waiting for error group managing segments... Aug 7 21:25:45.889551118 (11h46m7.633607968s) [upload=12186]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:45.889556440 (11h46m7.633613289s) [upload=12186]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:45.889559845 (11h46m7.633616705s) [upload=12186]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:15.619717489 (11h46m37.363774339s) [upload=12186]: Done uploading segments... Aug 7 21:26:15.619739739 (11h46m37.363796589s) [upload=12186]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:40.252456263 (11h47m1.996513132s) [upload=12201]: Starting upload Aug 7 21:26:40.252598168 (11h47m1.996655038s) [upload=12201]: Done starting upload Aug 7 21:26:40.252702071 (11h47m1.996758923s) [upload=12201]: Uploading segments... Aug 7 21:26:40.252782136 (11h47m1.996839003s) [upload=12201]: Got next segment. Inline: true Aug 7 21:26:40.252790092 (11h47m1.996846954s) [upload=12201]: Waiting for error group managing segments... Aug 7 21:26:40.252842463 (11h47m1.996899315s) [upload=12201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:40.252849740 (11h47m1.996906592s) [upload=12201]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:40.252854153 (11h47m1.996911015s) [upload=12201]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:09.981768433 (11h47m31.725825298s) [upload=12201]: Done uploading segments... Aug 7 21:27:09.981793631 (11h47m31.725850483s) [upload=12201]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:26.837217755 (11h47m48.581274619s) [upload=12211]: Starting upload Aug 7 21:27:26.837300223 (11h47m48.581357084s) [upload=12211]: Done starting upload Aug 7 21:27:26.837385984 (11h47m48.581442831s) [upload=12211]: Uploading segments... Aug 7 21:27:26.837444443 (11h47m48.581501313s) [upload=12211]: Got next segment. Inline: true Aug 7 21:27:26.837449770 (11h47m48.581506619s) [upload=12211]: Waiting for error group managing segments... Aug 7 21:27:26.837517257 (11h47m48.581574107s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:26.837523154 (11h47m48.581580003s) [upload=12211]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:26.837527097 (11h47m48.58158395s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:56.564982463 (11h48m18.309039313s) [upload=12211]: Done uploading segments... Aug 7 21:27:56.565004020 (11h48m18.309060871s) [upload=12211]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:36.367399060 (11h48m58.11145591s) [upload=12226]: Starting upload Aug 7 21:28:36.367588554 (11h48m58.111645417s) [upload=12226]: Done starting upload Aug 7 21:28:36.367664912 (11h48m58.111721765s) [upload=12226]: Uploading segments... Aug 7 21:28:36.367723895 (11h48m58.111780757s) [upload=12226]: Got next segment. Inline: true Aug 7 21:28:36.367729331 (11h48m58.111786192s) [upload=12226]: Waiting for error group managing segments... Aug 7 21:28:36.367785894 (11h48m58.111842754s) [upload=12226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:36.367791010 (11h48m58.11184787s) [upload=12226]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:36.367794492 (11h48m58.111851352s) [upload=12226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:06.092890267 (11h49m27.836947129s) [upload=12226]: Done uploading segments... Aug 7 21:29:06.092932205 (11h49m27.836989051s) [upload=12226]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:06.522310183 (11h49m28.266367033s) [upload=12233]: Starting upload Aug 7 21:29:06.522388772 (11h49m28.266445633s) [upload=12233]: Done starting upload Aug 7 21:29:06.522479144 (11h49m28.266535995s) [upload=12233]: Uploading segments... Aug 7 21:29:06.522560400 (11h49m28.266617269s) [upload=12233]: Got next segment. Inline: true Aug 7 21:29:06.522566995 (11h49m28.266623844s) [upload=12233]: Waiting for error group managing segments... Aug 7 21:29:06.522615251 (11h49m28.266672101s) [upload=12233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:06.522621757 (11h49m28.266678608s) [upload=12233]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:06.522630028 (11h49m28.266686879s) [upload=12233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:36.250046349 (11h49m57.994103211s) [upload=12233]: Done uploading segments... Aug 7 21:29:36.250088941 (11h49m57.994145808s) [upload=12233]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:03.947527959 (11h50m25.691584822s) [upload=12249]: Starting upload Aug 7 21:30:03.947767439 (11h50m25.691824311s) [upload=12249]: Done starting upload Aug 7 21:30:03.959515214 (11h50m25.703572066s) [upload=12249]: Uploading segments... Aug 7 21:30:03.959659046 (11h50m25.703715962s) [upload=12249]: Got next segment. Inline: true Aug 7 21:30:03.959668426 (11h50m25.703725302s) [upload=12249]: Waiting for error group managing segments... Aug 7 21:30:03.959783210 (11h50m25.703840082s) [upload=12249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:03.959790677 (11h50m25.703847531s) [upload=12249]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:03.959795320 (11h50m25.703852169s) [upload=12249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:33.655073335 (11h50m55.399130195s) [upload=12249]: Done uploading segments... Aug 7 21:30:33.655096560 (11h50m55.399153412s) [upload=12249]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:38.130475962 (11h50m59.874532829s) [upload=12256]: Starting upload Aug 7 21:30:38.130573915 (11h50m59.874630787s) [upload=12256]: Done starting upload Aug 7 21:30:38.130665890 (11h50m59.874722739s) [upload=12256]: Uploading segments... Aug 7 21:30:38.131017420 (11h50m59.875074302s) [upload=12256]: Got next segment. Inline: true Aug 7 21:30:38.131029217 (11h50m59.87508609s) [upload=12256]: Waiting for error group managing segments... Aug 7 21:30:38.131118811 (11h50m59.875175683s) [upload=12256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:38.131125270 (11h50m59.875182135s) [upload=12256]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:38.131131701 (11h50m59.875188569s) [upload=12256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:07.857077892 (11h51m29.601134744s) [upload=12256]: Done uploading segments... Aug 7 21:31:07.857100740 (11h51m29.601157585s) [upload=12256]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:11.159434284 (11h51m32.903491145s) [upload=12264]: Starting upload Aug 7 21:31:11.159626488 (11h51m32.903683349s) [upload=12264]: Done starting upload Aug 7 21:31:11.159711096 (11h51m32.903767956s) [upload=12264]: Uploading segments... Aug 7 21:31:11.159755148 (11h51m32.903812017s) [upload=12264]: Got next segment. Inline: true Aug 7 21:31:11.159760622 (11h51m32.903817485s) [upload=12264]: Waiting for error group managing segments... Aug 7 21:31:11.159814091 (11h51m32.903870951s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:11.159819842 (11h51m32.903876703s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:11.159823678 (11h51m32.903880539s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:40.887250927 (11h52m2.631307774s) [upload=12264]: Done uploading segments... Aug 7 21:31:40.887302452 (11h52m2.631359314s) [upload=12264]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:45.153828960 (11h52m6.89788582s) [upload=12272]: Starting upload Aug 7 21:31:45.153940931 (11h52m6.897997802s) [upload=12272]: Done starting upload Aug 7 21:31:45.154020734 (11h52m6.898077597s) [upload=12272]: Uploading segments... Aug 7 21:31:45.154082030 (11h52m6.898138908s) [upload=12272]: Got next segment. Inline: true Aug 7 21:31:45.154088359 (11h52m6.898145223s) [upload=12272]: Waiting for error group managing segments... Aug 7 21:31:45.154142898 (11h52m6.898199759s) [upload=12272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:45.154150494 (11h52m6.898207362s) [upload=12272]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:45.154155049 (11h52m6.898211914s) [upload=12272]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:14.861389353 (11h52m36.605446232s) [upload=12272]: Done uploading segments... Aug 7 21:32:14.861441235 (11h52m36.605498102s) [upload=12272]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:32:34.523168733 (11h52m56.267225597s) [upload=12286]: Starting upload Aug 7 21:32:34.523264231 (11h52m56.267321089s) [upload=12286]: Done starting upload Aug 7 21:32:34.523343267 (11h52m56.267400122s) [upload=12286]: Uploading segments... Aug 7 21:32:34.523393734 (11h52m56.267450607s) [upload=12286]: Got next segment. Inline: true Aug 7 21:32:34.523400482 (11h52m56.267457354s) [upload=12286]: Waiting for error group managing segments... Aug 7 21:32:34.523450495 (11h52m56.267507356s) [upload=12286]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:34.523455899 (11h52m56.267512759s) [upload=12286]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:32:34.523555922 (11h52m56.267612773s) [upload=12286]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:33:04.247355762 (11h53m25.991412634s) [upload=12286]: Done uploading segments... Aug 7 21:33:04.247398685 (11h53m25.991455557s) [upload=12286]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000849318s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001045515s FAIL (2025-08-07 21:33:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001066336s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.........................................XXXXXXXXXXXXXXXXXXX] duration: 30.000929694s FAIL (2025-08-07 21:32:47 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.00095473s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 1m0.001019992s FAIL (2025-08-07 21:32:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00106294s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [................................X..XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000933137s FAIL (2025-08-07 21:32:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000948037s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 882.748123ms FAIL (2025-08-07 21:32:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== loading segments... Aug 7 21:21:33.493714811 (11h41m55.237771657s) [upload=12125]: Upload finished. err: <nil> Aug 7 21:22:47.998800022 (11h43m9.742856869s) [upload=12141]: Starting upload Aug 7 21:22:47.998849424 (11h43m9.742906283s) [upload=12141]: Done starting upload Aug 7 21:22:47.998919383 (11h43m9.742976233s) [upload=12141]: Uploading segments... Aug 7 21:22:47.999001069 (11h43m9.743057938s) [upload=12141]: Got next segment. Inline: true Aug 7 21:22:47.999007866 (11h43m9.743064718s) [upload=12141]: Waiting for error group managing segments... Aug 7 21:22:47.999066684 (11h43m9.743123533s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:47.999073371 (11h43m9.743130221s) [upload=12141]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:47.999077096 (11h43m9.743133947s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:48.164602559 (11h43m9.90865943s) [upload=12141]: Done uploading segments... Aug 7 21:22:48.164642998 (11h43m9.908699853s) [upload=12141]: Upload finished. err: <nil> Aug 7 21:23:35.990065388 (11h43m57.734122239s) [upload=12154]: Starting upload Aug 7 21:23:35.990143300 (11h43m57.734200146s) [upload=12154]: Done starting upload Aug 7 21:23:35.990217262 (11h43m57.734274108s) [upload=12154]: Uploading segments... Aug 7 21:23:35.990277036 (11h43m57.734333925s) [upload=12154]: Got next segment. Inline: true Aug 7 21:23:35.990283708 (11h43m57.734340569s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:23:35.990337388 (11h43m57.734394253s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:35.990343849 (11h43m57.734400714s) [upload=12154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:35.990347914 (11h43m57.734405001s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:36.150463692 (11h43m57.894520544s) [upload=12154]: Done uploading segments... Aug 7 21:23:36.150483857 (11h43m57.894540707s) [upload=12154]: Upload finished. err: <nil> Aug 7 21:24:13.830997960 (11h44m35.57505482s) [upload=12160]: Starting upload Aug 7 21:24:13.831049433 (11h44m35.575106295s) [upload=12160]: Done starting upload Aug 7 21:24:13.831151824 (11h44m35.575208678s) [upload=12160]: Uploading segments... Aug 7 21:24:13.831192510 (11h44m35.575249377s) [upload=12160]: Got next segment. Inline: true Aug 7 21:24:13.831197614 (11h44m35.575254475s) [upload=12160]: Waiting for error group managing segments... Aug 7 21:24:13.831253497 (11h44m35.57531036s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:13.831259191 (11h44m35.575316046s) [upload=12160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:13.831262688 (11h44m35.575319551s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:13.982951222 (11h44m35.727008092s) [upload=12160]: Done uploading segments... Aug 7 21:24:13.982975878 (11h44m35.727032727s) [upload=12160]: Upload finished. err: <nil> Aug 7 21:24:48.198960455 (11h45m9.943017328s) [upload=12172]: Starting upload Aug 7 21:24:48.199050501 (11h45m9.943107348s) [upload=12172]: Done starting upload Aug 7 21:24:48.199166478 (11h45m9.943223341s) [upload=12172]: Uploading segments... Aug 7 21:24:48.199284289 (11h45m9.943341159s) [upload=12172]: Got next segment. Inline: true Aug 7 21:24:48.199309162 (11h45m9.943366036s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:24:48.199360469 (11h45m9.943417336s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:48.199382302 (11h45m9.943439163s) [upload=12172]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:48.199386864 (11h45m9.943443734s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:48.353366598 (11h45m10.09742347s) [upload=12172]: Done uploading segments... Aug 7 21:24:48.353410465 (11h45m10.097467327s) [upload=12172]: Upload finished. err: <nil> Aug 7 21:25:56.601363668 (11h46m18.345420518s) [upload=12190]: Starting upload Aug 7 21:25:56.601416202 (11h46m18.345473063s) [upload=12190]: Done starting upload Aug 7 21:25:56.601478987 (11h46m18.345535837s) [upload=12190]: Uploading segments... Aug 7 21:25:56.601519567 (11h46m18.345576436s) [upload=12190]: Got next segment. Inline: true Aug 7 21:25:56.601524613 (11h46m18.345581463s) [upload=12190]: Waiting for error group managing segments... Aug 7 21:25:56.601564530 (11h46m18.345621381s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:56.601571562 (11h46m18.345628412s) [upload=12190]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:56.601574952 (11h46m18.345631806s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.289489478 (11h46m19.033546339s) [upload=12190]: Done uploading segments... Aug 7 21:25:57.289514011 (11h46m19.033570861s) [upload=12190]: Upload finished. err: <nil> Aug 7 21:26:32.814119549 (11h46m54.558176399s) [upload=12198]: Starting upload Aug 7 21:26:32.814178314 (11h46m54.558235164s) [upload=12198]: Done starting upload Aug 7 21:26:32.814230962 (11h46m54.558287814s) [upload=12198]: Uploading segments... Aug 7 21:26:32.814314990 (11h46m54.558371858s) [upload=12198]: Got next segment. Inline: true Aug 7 21:26:32.814321493 (11h46m54.558378353s) [upload=12198]: Waiting for error group managing segments... Aug 7 21:26:32.814387101 (11h46m54.558443951s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.814393685 (11h46m54.558450545s) [upload=12198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:32.814397318 (11h46m54.558454179s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:32.972796167 (11h46m54.716853027s) [upload=12198]: Done uploading segments... Aug 7 21:26:32.972815783 (11h46m54.716872632s) [upload=12198]: Upload finished. err: <nil> Aug 7 21:27:41.280232963 (11h48m3.024289826s) [upload=12214]: Starting upload Aug 7 21:27:41.280291948 (11h48m3.024348811s) [upload=12214]: Done starting upload Aug 7 21:27:41.280410716 (11h48m3.024467565s) [upload=12214]: Uploading segments... Aug 7 21:27:41.280478626 (11h48m3.024535487s) [upload=12214]: Got next segment. Inline: true Aug 7 21:27:41.280485326 (11h48m3.024542193s) [upload=12214]: Waiting for error group managing segments... Aug 7 21:27:41.280533304 (11h48m3.024590156s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:41.280538628 (11h48m3.024595488s) [upload=12214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:41.280542274 (11h48m3.02459914s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:41.433978896 (11h48m3.178035773s) [upload=12214]: Done uploading segments... Aug 7 21:27:41.434003733 (11h48m3.178060594s) [upload=12214]: Upload finished. err: <nil> Aug 7 21:28:53.851663561 (11h49m15.595720412s) [upload=12231]: Starting upload Aug 7 21:28:53.851736366 (11h49m15.595793226s) [upload=12231]: Done starting upload Aug 7 21:28:53.851804427 (11h49m15.595861277s) [upload=12231]: Uploading segments... Aug 7 21:28:53.851851609 (11h49m15.595908471s) [upload=12231]: Got next segment. Inline: true Aug 7 21:28:53.851857251 (11h49m15.595914111s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:28:53.851899816 (11h49m15.595956668s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:53.851905032 (11h49m15.595961892s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:53.851908856 (11h49m15.595965718s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:54.191384948 (11h49m15.935441808s) [upload=12231]: Done uploading segments... Aug 7 21:28:54.191405192 (11h49m15.935462045s) [upload=12231]: Upload finished. err: <nil> Aug 7 21:29:19.999054333 (11h49m41.74311118s) [upload=12237]: Starting upload Aug 7 21:29:19.999128859 (11h49m41.743185722s) [upload=12237]: Done starting upload Aug 7 21:29:19.999188080 (11h49m41.743244931s) [upload=12237]: Uploading segments... Aug 7 21:29:19.999327277 (11h49m41.74338415s) [upload=12237]: Got next segment. Inline: true Aug 7 21:29:19.999335397 (11h49m41.743392265s) [upload=12237]: Waiting for error group managing segments... Aug 7 21:29:19.999418844 (11h49m41.743475691s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:19.999426848 (11h49m41.743483712s) [upload=12237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:19.999430264 (11h49m41.743487128s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:21.165098561 (11h49m42.909155421s) [upload=12237]: Done uploading segments... Aug 7 21:29:21.165118654 (11h49m42.909175505s) [upload=12237]: Upload finished. err: <nil> Aug 7 21:29:59.841735510 (11h50m21.585792373s) [upload=12245]: Starting upload Aug 7 21:29:59.841811008 (11h50m21.585867874s) [upload=12245]: Done starting upload Aug 7 21:29:59.841889062 (11h50m21.585945909s) [upload=12245]: Uploading segments... Aug 7 21:29:59.841961319 (11h50m21.58601819s) [upload=12245]: Got next segment. Inline: true Aug 7 21:29:59.841966490 (11h50m21.586023336s) [upload=12245]: Waiting for error group managing segments... Aug 7 21:29:59.842009456 (11h50m21.586066303s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.842013678 (11h50m21.586070524s) [upload=12245]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.842016402 (11h50m21.586073249s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.937966407 (11h50m21.682023257s) [upload=12245]: Done uploading segments... Aug 7 21:29:59.937989326 (11h50m21.682046187s) [upload=12245]: Upload finished. err: metaclient: Too Many Requests Duration: 882.779203ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001720847s FAIL (2025-08-07 21:32:32 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001738144s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.................................................XXXXXXXXXXX] duration: 30.001730753s FAIL (2025-08-07 21:32:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== in upload segment. Aug 7 21:20:13.797043921 (11h40m35.541100788s) [noise=true , address=77.90.31.18:4805 , piece_id=HKKJRNDDMITSXEXDDMZ37ZCG5OFNG72PAEDN7TNWDHXZQYBSWIXA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.797051683 (11h40m35.541108546s) [noise=true , address=104.36.82.15:4257 , piece_id=46HGSNBRDPF3JZMTPWY3OPAGK3RXBGTZWXDDNWZQERV6QAYWJ3TA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.797383845 (11h40m35.541440712s) [noise=true , address=77.90.31.17:4772 , piece_id=EMV6OLIN7YSBQQN4SC4VJGWQGFKLPBQVVJZTTKUE5ELQDXMXPSSQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.797528063 (11h40m35.541584979s) [noise=true , address=77.90.31.18:4807 , piece_id=O3SDUB6BQB6CPYN4X7Z7GHUK5Y663IOBGE4S4MB5UBB46HCQMJLA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.797616630 (11h40m35.541673477s) [noise=true , address=45.91.116.4:4937 , piece_id=PGZP47Q6KAXPGOPJ5JAAKADDBRWMAIBX5GWL6JGHFJUU3AMOPKTQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.797841016 (11h40m35.541897882s) [noise=true , address=45.91.116.10:4082 , piece_id=3AL7IMQKZU26B34QQEK43SAXSWT6W7M4TF4ITTYF774U2LED2LRQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.797890836 (11h40m35.541947696s) [noise=true , address=45.91.116.6:4973 , piece_id=UKM3AZMNW67WQ7WMFP5Z55NG4JBZBFLDVODUHIISQYZ27HUVHP6Q, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798097154 (11h40m35.542154017s) [noise=true , address=45.91.116.8:4047 , piece_id=RCBP5UBGZ5C6YEQQQ3SD3GUU26LJTFX3GOL4XZOKEFZOXB4WDRSA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798331623 (11h40m35.542388488s) [noise=true , address=104.36.82.7:4912 , piece_id=2XMUVIQB6QCEDOJGJR7CY6FW6PALYZMQRA7KNPSQQB4IAYHPQZRA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798662439 (11h40m35.542719308s) [noise=true , address=77.90.31.13:4627 , piece_id=TOKP44CMWL4OXYLWOFHL7WFKSN2C57WGHWBPZMQDVZ4GRDKLCBRA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798759384 (11h40m35.54281625s) [noise=true , address=77.90.31.10:4519 , piece_id=FDV33ZMCVYJOMT4YP7KIRA3MBB7OUNJHAPITYYLMXJW6DZWKGGCQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798971326 (11h40m35.54302818s) [noise=true , address=45.91.116.12:4225 , piece_id=EHBNQPDYM7HB4OGRJGU6QZ3CTM5YWHUEV2SSD5LKUZI6LGG6X4GA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.799201622 (11h40m35.543258483s) [noise=true , address=104.36.82.2:4731 , piece_id=DKWMN2CLFT3XVYTGEF3HJYZXPME5TE37I45TD5ABWVRQMWE3X6YA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.799847282 (11h40m35.543904151s) [noise=true , address=77.90.31.13:4626 , piece_id=VNGCTMK7ET2RCVPEDCDXZHYKARZIIGUU6CPVT6LKA7HCLWJFJ7AQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.800536275 (11h40m35.544593146s) [noise=true , address=45.91.116.5:4297 , piece_id=77LOHOMR6FRLTTYGTAVC4SHMSDSDWGNITJMIIKDIA7ADS2272TAQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.800663553 (11h40m35.544720423s) [noise=true , address=45.91.116.12:4226 , piece_id=47PAAFR7VRN7H7MBS7Z7KRONLGN6Q2JF5CBYT6ASUIANYBUSYNKQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:14.404668625 (11h40m36.148725475s) [upload=12096]: Next returned nil segment Aug 7 21:20:14.404685815 (11h40m36.148742676s) [upload=12096]: Waiting for error group managing segments... Aug 7 21:20:14.561430060 (11h40m36.305486911s) [noise=true , address=45.91.116.12:4225 , piece_id=EHBNQPDYM7HB4OGRJGU6QZ3CTM5YWHUEV2SSD5LKUZI6LGG6X4GA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.583124069 (11h40m36.327180919s) [noise=true , address=104.36.82.15:4257 , piece_id=46HGSNBRDPF3JZMTPWY3OPAGK3RXBGTZWXDDNWZQERV6QAYWJ3TA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.622563656 (11h40m36.366620519s) [noise=true , address=45.91.116.8:4047 , piece_id=RCBP5UBGZ5C6YEQQQ3SD3GUU26LJTFX3GOL4XZOKEFZOXB4WDRSA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.648162660 (11h40m36.392219508s) [noise=true , address=77.90.31.17:4772 , piece_id=EMV6OLIN7YSBQQN4SC4VJGWQGFKLPBQVVJZTTKUE5ELQDXMXPSSQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.649278924 (11h40m36.393335781s) [noise=true , address=104.36.82.7:4912 , piece_id=2XMUVIQB6QCEDOJGJR7CY6FW6PALYZMQRA7KNPSQQB4IAYHPQZRA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.657067139 (11h40m36.401123988s) [noise=true , address=77.90.31.18:4807 , piece_id=O3SDUB6BQB6CPYN4X7Z7GHUK5Y663IOBGE4S4MB5UBB46HCQMJLA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.660665476 (11h40m36.404722337s) [noise=true , address=77.90.31.13:4627 , piece_id=TOKP44CMWL4OXYLWOFHL7WFKSN2C57WGHWBPZMQDVZ4GRDKLCBRA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.692922160 (11h40m36.43697902s) [noise=true , address=104.36.82.2:4731 , piece_id=DKWMN2CLFT3XVYTGEF3HJYZXPME5TE37I45TD5ABWVRQMWE3X6YA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.693482368 (11h40m36.437539243s) [noise=true , address=45.91.116.12:4226 , piece_id=47PAAFR7VRN7H7MBS7Z7KRONLGN6Q2JF5CBYT6ASUIANYBUSYNKQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.722004664 (11h40m36.466061524s) [noise=true , address=45.91.116.5:4297 , piece_id=77LOHOMR6FRLTTYGTAVC4SHMSDSDWGNITJMIIKDIA7ADS2272TAQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.747287536 (11h40m36.491344386s) [noise=true , address=45.91.116.6:4973 , piece_id=UKM3AZMNW67WQ7WMFP5Z55NG4JBZBFLDVODUHIISQYZ27HUVHP6Q, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.751701687 (11h40m36.495758555s) [noise=true , address=77.90.31.18:4805 , piece_id=HKKJRNDDMITSXEXDDMZ37ZCG5OFNG72PAEDN7TNWDHXZQYBSWIXA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.798490539 (11h40m36.542547386s) [noise=true , address=77.90.31.10:4519 , piece_id=FDV33ZMCVYJOMT4YP7KIRA3MBB7OUNJHAPITYYLMXJW6DZWKGGCQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.798534450 (11h40m36.54259131s) [seg_pos={0 0}, upload=12096]: Segment reached optimal threshold of 13 pieces. Aug 7 21:20:14.798673827 (11h40m36.542730687s) [noise=true , address=45.91.116.10:4082 , piece_id=3AL7IMQKZU26B34QQEK43SAXSWT6W7M4TF4ITTYF774U2LED2LRQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798750267 (11h40m36.542807123s) [noise=true , address=77.90.31.13:4626 , piece_id=VNGCTMK7ET2RCVPEDCDXZHYKARZIIGUU6CPVT6LKA7HCLWJFJ7AQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798798579 (11h40m36.542855426s) [noise=true , address=45.91.116.4:4937 , piece_id=PGZP47Q6KAXPGOPJ5JAAKADDBRWMAIBX5GWL6JGHFJUU3AMOPKTQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798820360 (11h40m36.54287721s) [seg_pos={0 0}, upload=12096]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:20:14.798910105 (11h40m36.542966966s) [upload=12096]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:20:14.798915686 (11h40m36.542972547s) [upload=12096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:24.464709964 (11h51m46.208766834s) [upload=12096]: Done uploading segments... Aug 7 21:31:24.464745783 (11h51m46.208802653s) [upload=12096]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:24.881904902 (11h51m46.625961766s) [upload=12268]: Starting upload Aug 7 21:31:24.882094151 (11h51m46.626151016s) [upload=12268]: Done starting upload Aug 7 21:31:24.882164223 (11h51m46.626221082s) [upload=12268]: Uploading segments... Aug 7 21:31:24.882475408 (11h51m46.626532277s) [upload=12268]: Got next segment. Inline: false Aug 7 21:31:24.882552261 (11h51m46.626609127s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:24.882560165 (11h51m46.626617032s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:54.614102730 (11h52m16.358159588s) [upload=12268]: Next segment err: context deadline exceeded Aug 7 21:31:54.614205863 (11h52m16.358262714s) [upload=12268]: Done uploading segments... Aug 7 21:31:54.614225924 (11h52m16.358282773s) [upload=12268]: Upload finished. err: context deadline exceeded Aug 7 21:32:00.253568814 (11h52m21.997625667s) [upload=12278]: Starting upload Aug 7 21:32:00.253669214 (11h52m21.997726066s) [upload=12278]: Done starting upload Aug 7 21:32:00.254008956 (11h52m21.998065808s) [upload=12278]: Uploading segments... Aug 7 21:32:00.254068254 (11h52m21.998125125s) [upload=12278]: Got next segment. Inline: false Aug 7 21:32:00.254112779 (11h52m21.998169639s) [upload=12278]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:32:00.254118157 (11h52m21.998175018s) [upload=12278]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:29.984781139 (11h52m51.728838008s) [upload=12278]: Next segment err: context deadline exceeded Aug 7 21:32:29.984827795 (11h52m51.728884645s) [upload=12278]: Done uploading segments... Aug 7 21:32:29.984851910 (11h52m51.728908763s) [upload=12278]: Upload finished. err: context deadline exceeded Duration: 30.001754865s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 17 out of 60 is failed (threshold=3) [..........................................X.XXXXXXXXXXXXXXXX] duration: 1m0.00107475s FAIL (2025-08-07 21:32:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [upload=12134]: Uploading segments... Aug 7 21:22:14.360380199 (11h42m36.104437068s) [upload=12134]: Got next segment. Inline: true Aug 7 21:22:14.360385659 (11h42m36.10444252s) [upload=12134]: Waiting for error group managing segments... Aug 7 21:22:14.360431265 (11h42m36.104488116s) [upload=12134]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:14.360437033 (11h42m36.104493884s) [upload=12134]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:14.360440534 (11h42m36.104497395s) [upload=12134]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:14.088328715 (11h43m35.832385565s) [upload=12134]: Done uploading segments... Aug 7 21:23:14.088350462 (11h43m35.832407323s) [upload=12134]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:23:14.505772185 (11h43m36.249829051s) [upload=12148]: Starting upload Aug 7 21:23:14.505877531 (11h43m36.249934403s) [upload=12148]: Done starting upload Aug 7 21:23:14.505973975 (11h43m36.250030823s) [upload=12148]: Uploading segments... Aug 7 21:23:14.506132768 (11h43m36.250189638s) [upload=12148]: Got next segment. Inline: true Aug 7 21:23:14.506140538 (11h43m36.250197403s) [upload=12148]: Waiting for error group managing segments... Aug 7 21:23:14.506211738 (11h43m36.250268603s) [upload=12148]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:14.506218667 (11h43m36.250275531s) [upload=12148]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:14.506222979 (11h43m36.250279842s) [upload=12148]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:14.230441064 (11h44m35.974497931s) [upload=12148]: Done uploading segments... Aug 7 21:24:14.230468679 (11h44m35.974525551s) [upload=12148]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:24:14.664138591 (11h44m36.408195463s) [upload=12162]: Starting upload Aug 7 21:24:14.664235374 (11h44m36.408292237s) [upload=12162]: Done starting upload Aug 7 21:24:14.664324426 (11h44m36.408381273s) [upload=12162]: Uploading segments... Aug 7 21:24:14.664378438 (11h44m36.408435308s) [upload=12162]: Got next segment. Inline: true Aug 7 21:24:14.664385109 (11h44m36.408441977s) [upload=12162]: Waiting for error group managing segments... Aug 7 21:24:14.664444486 (11h44m36.408501361s) [upload=12162]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:14.664452219 (11h44m36.408509087s) [upload=12162]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:14.664455983 (11h44m36.40851285s) [upload=12162]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:14.390008254 (11h45m36.134065107s) [upload=12162]: Done uploading segments... Aug 7 21:25:14.390037098 (11h45m36.134093947s) [upload=12162]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:14.891337243 (11h45m36.635394094s) [upload=12178]: Starting upload Aug 7 21:25:14.891405973 (11h45m36.635462834s) [upload=12178]: Done starting upload Aug 7 21:25:14.891587260 (11h45m36.635644108s) [upload=12178]: Uploading segments... Aug 7 21:25:14.891659588 (11h45m36.635716456s) [upload=12178]: Got next segment. Inline: true Aug 7 21:25:14.891666181 (11h45m36.635723042s) [upload=12178]: Waiting for error group managing segments... Aug 7 21:25:14.891713596 (11h45m36.635770446s) [upload=12178]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:14.891719665 (11h45m36.635776525s) [upload=12178]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:14.891723701 (11h45m36.635780562s) [upload=12178]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:14.618805682 (11h46m36.362862552s) [upload=12178]: Done uploading segments... Aug 7 21:26:14.618848987 (11h46m36.36290586s) [upload=12178]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:15.042634192 (11h46m36.786691042s) [upload=12195]: Starting upload Aug 7 21:26:15.042707477 (11h46m36.786764348s) [upload=12195]: Done starting upload Aug 7 21:26:15.042793174 (11h46m36.786850024s) [upload=12195]: Uploading segments... Aug 7 21:26:15.042848386 (11h46m36.78690526s) [upload=12195]: Got next segment. Inline: true Aug 7 21:26:15.042855793 (11h46m36.786912664s) [upload=12195]: Waiting for error group managing segments... Aug 7 21:26:15.042911164 (11h46m36.786968025s) [upload=12195]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:15.042918440 (11h46m36.786975312s) [upload=12195]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:15.042924061 (11h46m36.78698093s) [upload=12195]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:14.771724725 (11h47m36.515781576s) [upload=12195]: Done uploading segments... Aug 7 21:27:14.771758030 (11h47m36.515814885s) [upload=12195]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:15.202352221 (11h47m36.946409081s) [upload=12209]: Starting upload Aug 7 21:27:15.202440356 (11h47m36.946497217s) [upload=12209]: Done starting upload Aug 7 21:27:15.202525888 (11h47m36.946582739s) [upload=12209]: Uploading segments... Aug 7 21:27:15.202588330 (11h47m36.946645196s) [upload=12209]: Got next segment. Inline: true Aug 7 21:27:15.202594295 (11h47m36.946651156s) [upload=12209]: Waiting for error group managing segments... Aug 7 21:27:15.202651093 (11h47m36.946707944s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:15.202656763 (11h47m36.946713624s) [upload=12209]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:15.202660199 (11h47m36.946717059s) [upload=12209]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:14.931851536 (11h48m36.675908396s) [upload=12209]: Done uploading segments... Aug 7 21:28:14.931874519 (11h48m36.67593138s) [upload=12209]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:21.072819157 (11h48m42.816876017s) [upload=12223]: Starting upload Aug 7 21:28:21.072899208 (11h48m42.816956068s) [upload=12223]: Done starting upload Aug 7 21:28:21.072981263 (11h48m42.817038113s) [upload=12223]: Uploading segments... Aug 7 21:28:21.073042235 (11h48m42.817099103s) [upload=12223]: Got next segment. Inline: true Aug 7 21:28:21.073047353 (11h48m42.817104214s) [upload=12223]: Waiting for error group managing segments... Aug 7 21:28:21.073104067 (11h48m42.817160927s) [upload=12223]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:21.073109154 (11h48m42.817166016s) [upload=12223]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:21.073112378 (11h48m42.817169238s) [upload=12223]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:20.797200985 (11h49m42.541258063s) [upload=12223]: Done uploading segments... Aug 7 21:29:20.797233820 (11h49m42.541290687s) [upload=12223]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:21.230279976 (11h49m42.974336838s) [upload=12238]: Starting upload Aug 7 21:29:21.230391607 (11h49m42.974448467s) [upload=12238]: Done starting upload Aug 7 21:29:21.230482429 (11h49m42.974539278s) [upload=12238]: Uploading segments... Aug 7 21:29:21.230606006 (11h49m42.974662867s) [upload=12238]: Got next segment. Inline: true Aug 7 21:29:21.230613609 (11h49m42.974670469s) [upload=12238]: Waiting for error group managing segments... Aug 7 21:29:21.230660759 (11h49m42.974717609s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:21.230666510 (11h49m42.974723374s) [upload=12238]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:21.230691185 (11h49m42.974748046s) [upload=12238]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:20.959347392 (11h50m42.703404243s) [upload=12238]: Done uploading segments... Aug 7 21:30:20.959370210 (11h50m42.70342706s) [upload=12238]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:21.398814384 (11h50m43.142871245s) [upload=12254]: Starting upload Aug 7 21:30:21.398880649 (11h50m43.142937512s) [upload=12254]: Done starting upload Aug 7 21:30:21.398949225 (11h50m43.143006078s) [upload=12254]: Uploading segments... Aug 7 21:30:21.398993888 (11h50m43.143050757s) [upload=12254]: Got next segment. Inline: true Aug 7 21:30:21.399003181 (11h50m43.143060046s) [upload=12254]: Waiting for error group managing segments... Aug 7 21:30:21.399046484 (11h50m43.143103345s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:21.399052138 (11h50m43.143108999s) [upload=12254]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:21.399055744 (11h50m43.143112604s) [upload=12254]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:21.120945605 (11h51m42.865002833s) [upload=12254]: Done uploading segments... Aug 7 21:31:21.121002610 (11h51m42.865059457s) [upload=12254]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:21.580865540 (11h51m43.324922404s) [upload=12267]: Starting upload Aug 7 21:31:21.580962495 (11h51m43.32501936s) [upload=12267]: Done starting upload Aug 7 21:31:21.581042226 (11h51m43.325099087s) [upload=12267]: Uploading segments... Aug 7 21:31:21.581118360 (11h51m43.32517523s) [upload=12267]: Got next segment. Inline: true Aug 7 21:31:21.581124116 (11h51m43.325180981s) [upload=12267]: Waiting for error group managing segments... Aug 7 21:31:21.581178813 (11h51m43.325235677s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:21.581184674 (11h51m43.325241538s) [upload=12267]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:21.581188187 (11h51m43.325245048s) [upload=12267]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:21.285242448 (11h52m43.029299295s) [upload=12267]: Done uploading segments... Aug 7 21:32:21.285264804 (11h52m43.029321651s) [upload=12267]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001089704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 22 out of 60 is failed (threshold=3) [......................................XXXXXXXXXXXXXXXXXXXXXX] duration: 30.001684142s FAIL (2025-08-07 21:32:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001712866s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X......................X.......................X] duration: 1.860852021s FAIL (2025-08-07 21:32:17 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M ======LOGS====== =12281]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:32:16.531340011 (11h52m38.275396872s) [seg_pos={0 0}, upload=12281]: Begin upload segment... Aug 7 21:32:16.531424068 (11h52m38.27548092s) [seg_pos={0 0}, upload=12281]: Done begin upload segment. Aug 7 21:32:16.531447266 (11h52m38.275504117s) [noise=true , address=104.36.82.15:4274 , piece_id=AYYRHJUYBOHZCJNYYK3XRKQD6RF4GGH3XH2BWXFEWPQQGIFS3LXA, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.531576038 (11h52m38.2756329s) [noise=true , address=45.91.116.12:4250 , piece_id=4M3RVR4XS7MIYCXSKQX5FI55T75I7SQFSPSBLOMRZQP6NS76HYWQ, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.531971743 (11h52m38.276028609s) [noise=true , address=104.36.82.44:4890 , piece_id=XJRASBP2J6AN4T3FYRATD47XQV2XXOTYGUV6CYNJEVG4CJCXBIQA, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.532061879 (11h52m38.276118749s) [noise=true , address=104.36.82.18:4367 , piece_id=PSZLVOTQKRKOUIQ7RSUT67EPUIG23L634M2XPGEKMB5BQXANKFTA, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.534099751 (11h52m38.278156627s) [noise=true , address=104.36.82.23:4453 , piece_id=GRHG6RZHB2SAL4E3NJAD22VV67C5P6JEYBJZ5GFCWOJFD7QWJY2Q, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.534534681 (11h52m38.278591551s) [noise=true , address=104.36.82.55:4206 , piece_id=TUX2FD43EYVQBXDUIEGB2JKG675BRQAXY4ADJUP5Z3FPVF3PMSKA, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.535016671 (11h52m38.279073541s) [noise=true , address=104.36.82.56:4239 , piece_id=TEMAUICTH3YP6CSIVCWMC4DQTAKJJGRIF3PKXZJPFMFAAA6JILLA, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.535295961 (11h52m38.279352822s) [noise=true , address=104.36.82.13:4209 , piece_id=HGRX4FBVYNIHZI3IGOHFAQ2GS5F7ALKHDP4VGQYDK5BDVJ33E23A, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.535745902 (11h52m38.279802771s) [noise=true , address=104.36.82.43:4857 , piece_id=EKSZXKJJE3GVVJOTRQBVOOOKED2GDXEXEHWE7TM6XJW2VIEHOS3A, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.536103453 (11h52m38.280160319s) [noise=true , address=64.49.13.139:4479 , piece_id=AVSIZTRK2IFYYFD3ULP7ANY3IRWPLJGQE57YN7OOD5HCBMUREGQA, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.536461791 (11h52m38.280518654s) [noise=true , address=104.36.82.7:4924 , piece_id=5GAWE7PORFN3MKVDB7VNNQIFZ2YNVYUIGT3AI5IP7PEFVXKBLXXA, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.536764078 (11h52m38.28082094s) [noise=true , address=104.36.82.38:4705 , piece_id=PUTP643WSHIOTQJUR6DC2AVD2NQIT5UOJWEOEXL4RT34PG4TPVSQ, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.537033209 (11h52m38.28109007s) [noise=true , address=45.91.116.11:4129 , piece_id=2PXIP4IUA64SDCLGNFEUGCMPNIL6DXNSKARRA3L3MN2DOTX4USZQ, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.537311621 (11h52m38.281368481s) [noise=true , address=104.36.82.16:4320 , piece_id=F7DZMOPDRWLU3LQXHZY7BWL2O66EN7Y4KCFSPXTX6LE6ZSY7ESCQ, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.537579670 (11h52m38.281636533s) [noise=true , address=104.36.82.9:4011 , piece_id=W3DFPOX3SM6AEJ3ISVX763UR6PEMEDXGNLNQOFJWDFRVFYBYBTRQ, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.537851259 (11h52m38.281908123s) [noise=true , address=104.36.82.39:4753 , piece_id=BSOBV5GACIR7MEJSMTL625DU5TLHDDBSHMS37XW3KQ3NF7OTTEFQ, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.538156341 (11h52m38.282213201s) [noise=true , address=104.36.82.37:4674 , piece_id=4P7SCLAB7EQMDICV6DPTXGS2GNYY22D2DPBRMSIBVF6BHQVX2SYQ, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:16.538504600 (11h52m38.282561535s) [noise=true , address=104.36.82.58:4265 , piece_id=HNFPBVUKWFYDI547AM7EIBW5ZNTF7EJWFLEZLXSFPG2RBTMHSSHQ, seg_pos={0 0}, upload=12281]: Uploading piece... Aug 7 21:32:17.109196479 (11h52m38.853253325s) [upload=12281]: Next returned nil segment Aug 7 21:32:17.109213409 (11h52m38.853270269s) [upload=12281]: Waiting for error group managing segments... Aug 7 21:32:17.263102675 (11h52m39.007159526s) [noise=true , address=45.91.116.12:4250 , piece_id=4M3RVR4XS7MIYCXSKQX5FI55T75I7SQFSPSBLOMRZQP6NS76HYWQ, seg_pos={0 0}, upload=12281]: Done uploading piece. err: <nil> Aug 7 21:32:17.296992590 (11h52m39.041049444s) [noise=true , address=104.36.82.18:4367 , piece_id=PSZLVOTQKRKOUIQ7RSUT67EPUIG23L634M2XPGEKMB5BQXANKFTA, seg_pos={0 0}, upload=12281]: Done uploading piece. err: <nil> Aug 7 21:32:17.300345526 (11h52m39.044402384s) [noise=true , address=45.91.116.11:4129 , piece_id=2PXIP4IUA64SDCLGNFEUGCMPNIL6DXNSKARRA3L3MN2DOTX4USZQ, seg_pos={0 0}, upload=12281]: Done uploading piece. err: <nil> Aug 7 21:32:17.316919884 (11h52m39.060976751s) [noise=true , address=104.36.82.39:4753 , piece_id=BSOBV5GACIR7MEJSMTL625DU5TLHDDBSHMS37XW3KQ3NF7OTTEFQ, seg_pos={0 0}, upload=12281]: Done uploading piece. err: <nil> Aug 7 21:32:17.339043479 (11h52m39.083100339s) [noise=true , address=104.36.82.55:4206 , piece_id=TUX2FD43EYVQBXDUIEGB2JKG675BRQAXY4ADJUP5Z3FPVF3PMSKA, seg_pos={0 0}, upload=12281]: Done uploading piece. err: <nil> Aug 7 21:32:17.339214210 (11h52m39.083271075s) [noise=true , address=104.36.82.58:4265 , piece_id=HNFPBVUKWFYDI547AM7EIBW5ZNTF7EJWFLEZLXSFPG2RBTMHSSHQ, seg_pos={0 0}, upload=12281]: Done uploading piece. err: <nil> Aug 7 21:32:17.344719591 (11h52m39.088776445s) [noise=true , address=104.36.82.7:4924 , piece_id=5GAWE7PORFN3MKVDB7VNNQIFZ2YNVYUIGT3AI5IP7PEFVXKBLXXA, seg_pos={0 0}, upload=12281]: Done uploading piece. err: <nil> Aug 7 21:32:17.349892785 (11h52m39.093949635s) [noise=true , address=104.36.82.13:4209 , piece_id=HGRX4FBVYNIHZI3IGOHFAQ2GS5F7ALKHDP4VGQYDK5BDVJ33E23A, seg_pos={0 0}, upload=12281]: Done uploading piece. err: <nil> Aug 7 21:32:17.381092652 (11h52m39.125149502s) [noise=true , address=104.36.82.23:4453 , piece_id=GRHG6RZHB2SAL4E3NJAD22VV67C5P6JEYBJZ5GFCWOJFD7QWJY2Q, seg_pos={0 0}, upload=12281]: Done uploading piece. err: <nil> Aug 7 21:32:17.386776008 (11h52m39.130832876s) [noise=true , address=104.36.82.38:4705 , piece_id=PUTP643WSHIOTQJUR6DC2AVD2NQIT5UOJWEOEXL4RT34PG4TPVSQ, seg_pos={0 0}, upload=12281]: Done uploading piece. err: <nil> Aug 7 21:32:17.406809981 (11h52m39.150866831s) [noise=true , address=104.36.82.15:4274 , piece_id=AYYRHJUYBOHZCJNYYK3XRKQD6RF4GGH3XH2BWXFEWPQQGIFS3LXA, seg_pos={0 0}, upload=12281]: Done uploading piece. err: <nil> Aug 7 21:32:17.434388458 (11h52m39.17844531s) [noise=true , address=104.36.82.56:4239 , piece_id=TEMAUICTH3YP6CSIVCWMC4DQTAKJJGRIF3PKXZJPFMFAAA6JILLA, seg_pos={0 0}, upload=12281]: Done uploading piece. err: <nil> Aug 7 21:32:17.521742710 (11h52m39.26579956s) [noise=true , address=104.36.82.44:4890 , piece_id=XJRASBP2J6AN4T3FYRATD47XQV2XXOTYGUV6CYNJEVG4CJCXBIQA, seg_pos={0 0}, upload=12281]: Done uploading piece. err: <nil> Aug 7 21:32:17.569377834 (11h52m39.313434695s) [noise=true , address=104.36.82.16:4320 , piece_id=F7DZMOPDRWLU3LQXHZY7BWL2O66EN7Y4KCFSPXTX6LE6ZSY7ESCQ, seg_pos={0 0}, upload=12281]: Done uploading piece. err: <nil> Aug 7 21:32:17.569416390 (11h52m39.313473252s) [seg_pos={0 0}, upload=12281]: Segment reached optimal threshold of 14 pieces. Aug 7 21:32:17.569823793 (11h52m39.313880643s) [noise=true , address=64.49.13.139:4479 , piece_id=AVSIZTRK2IFYYFD3ULP7ANY3IRWPLJGQE57YN7OOD5HCBMUREGQA, seg_pos={0 0}, upload=12281]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fa8o8SX7mogECREbm3fLwNDTNT4qVGSa8NUwaG1GkNe96FFqw): protocol: read tcp 139.59.144.22:46946->64.49.13.139:4479: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:32:17.569843607 (11h52m39.313900472s) [noise=true , address=104.36.82.37:4674 , piece_id=4P7SCLAB7EQMDICV6DPTXGS2GNYY22D2DPBRMSIBVF6BHQVX2SYQ, seg_pos={0 0}, upload=12281]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ufd6LJVsd8aX1LoHCnBfS9JYsvZTYyVgca48fNif9bx639iYX): protocol: write tcp 139.59.144.22:51502->104.36.82.37:4674: use of closed network connection; write tcp 139.59.144.22:51502->104.36.82.37:4674: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:51502->104.36.82.37:4674: use of closed network connection Aug 7 21:32:17.570038710 (11h52m39.314095564s) [noise=true , address=104.36.82.43:4857 , piece_id=EKSZXKJJE3GVVJOTRQBVOOOKED2GDXEXEHWE7TM6XJW2VIEHOS3A, seg_pos={0 0}, upload=12281]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DK5cJhprAFbNCSceZ527v5rMFw7e3teUTsxBXVPcjXUDiVv4P): protocol: read tcp 139.59.144.22:59664->104.36.82.43:4857: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:32:17.570084291 (11h52m39.31414116s) [noise=true , address=104.36.82.9:4011 , piece_id=W3DFPOX3SM6AEJ3ISVX763UR6PEMEDXGNLNQOFJWDFRVFYBYBTRQ, seg_pos={0 0}, upload=12281]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J8J47emDh4p115m69EqUYroi9GgruC3y8gmv2xVULVJSvFYYP): protocol: read tcp 139.59.144.22:40648->104.36.82.9:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 21:32:17.570117531 (11h52m39.31417438s) [seg_pos={0 0}, upload=12281]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:32:17.570207545 (11h52m39.314264406s) [upload=12281]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:32:17.570214958 (11h52m39.314271819s) [upload=12281]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:17.715142154 (11h52m39.459199003s) [upload=12281]: Done uploading segments... Aug 7 21:32:17.715159411 (11h52m39.459216258s) [upload=12281]: Upload finished. err: <nil> Duration:* 1.860880052s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..........................................XXXXXXXXXXXXXXXXXX] duration: 30.000756534s FAIL (2025-08-07 21:32:16 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: context deadline exceeded Duration: 30.000785478s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 26 out of 60 is failed (threshold=3) [.................................XX.XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001068754s FAIL (2025-08-07 21:32:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 21258 (11h45m37.504178104s) [upload=12180]: Uploading segments... Aug 7 21:25:15.760159734 (11h45m37.504216604s) [upload=12180]: Got next segment. Inline: true Aug 7 21:25:15.760164169 (11h45m37.504221029s) [upload=12180]: Waiting for error group managing segments... Aug 7 21:25:15.760202724 (11h45m37.504259584s) [upload=12180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:15.760207148 (11h45m37.504264007s) [upload=12180]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:15.760212108 (11h45m37.504268971s) [upload=12180]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:45.477193191 (11h46m7.22125006s) [upload=12180]: Done uploading segments... Aug 7 21:25:45.477222897 (11h46m7.22127976s) [upload=12180]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:45.889252639 (11h46m7.633309501s) [upload=12186]: Starting upload Aug 7 21:25:45.889337905 (11h46m7.633394765s) [upload=12186]: Done starting upload Aug 7 21:25:45.889442879 (11h46m7.633499729s) [upload=12186]: Uploading segments... Aug 7 21:25:45.889501484 (11h46m7.633558347s) [upload=12186]: Got next segment. Inline: true Aug 7 21:25:45.889507484 (11h46m7.633564344s) [upload=12186]: Waiting for error group managing segments... Aug 7 21:25:45.889551118 (11h46m7.633607968s) [upload=12186]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:45.889556440 (11h46m7.633613289s) [upload=12186]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:45.889559845 (11h46m7.633616705s) [upload=12186]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:15.619717489 (11h46m37.363774339s) [upload=12186]: Done uploading segments... Aug 7 21:26:15.619739739 (11h46m37.363796589s) [upload=12186]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:40.252456263 (11h47m1.996513132s) [upload=12201]: Starting upload Aug 7 21:26:40.252598168 (11h47m1.996655038s) [upload=12201]: Done starting upload Aug 7 21:26:40.252702071 (11h47m1.996758923s) [upload=12201]: Uploading segments... Aug 7 21:26:40.252782136 (11h47m1.996839003s) [upload=12201]: Got next segment. Inline: true Aug 7 21:26:40.252790092 (11h47m1.996846954s) [upload=12201]: Waiting for error group managing segments... Aug 7 21:26:40.252842463 (11h47m1.996899315s) [upload=12201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:40.252849740 (11h47m1.996906592s) [upload=12201]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:40.252854153 (11h47m1.996911015s) [upload=12201]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:09.981768433 (11h47m31.725825298s) [upload=12201]: Done uploading segments... Aug 7 21:27:09.981793631 (11h47m31.725850483s) [upload=12201]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:26.837217755 (11h47m48.581274619s) [upload=12211]: Starting upload Aug 7 21:27:26.837300223 (11h47m48.581357084s) [upload=12211]: Done starting upload Aug 7 21:27:26.837385984 (11h47m48.581442831s) [upload=12211]: Uploading segments... Aug 7 21:27:26.837444443 (11h47m48.581501313s) [upload=12211]: Got next segment. Inline: true Aug 7 21:27:26.837449770 (11h47m48.581506619s) [upload=12211]: Waiting for error group managing segments... Aug 7 21:27:26.837517257 (11h47m48.581574107s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:26.837523154 (11h47m48.581580003s) [upload=12211]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:26.837527097 (11h47m48.58158395s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:56.564982463 (11h48m18.309039313s) [upload=12211]: Done uploading segments... Aug 7 21:27:56.565004020 (11h48m18.309060871s) [upload=12211]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:36.367399060 (11h48m58.11145591s) [upload=12226]: Starting upload Aug 7 21:28:36.367588554 (11h48m58.111645417s) [upload=12226]: Done starting upload Aug 7 21:28:36.367664912 (11h48m58.111721765s) [upload=12226]: Uploading segments... Aug 7 21:28:36.367723895 (11h48m58.111780757s) [upload=12226]: Got next segment. Inline: true Aug 7 21:28:36.367729331 (11h48m58.111786192s) [upload=12226]: Waiting for error group managing segments... Aug 7 21:28:36.367785894 (11h48m58.111842754s) [upload=12226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:36.367791010 (11h48m58.11184787s) [upload=12226]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:36.367794492 (11h48m58.111851352s) [upload=12226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:06.092890267 (11h49m27.836947129s) [upload=12226]: Done uploading segments... Aug 7 21:29:06.092932205 (11h49m27.836989051s) [upload=12226]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:06.522310183 (11h49m28.266367033s) [upload=12233]: Starting upload Aug 7 21:29:06.522388772 (11h49m28.266445633s) [upload=12233]: Done starting upload Aug 7 21:29:06.522479144 (11h49m28.266535995s) [upload=12233]: Uploading segments... Aug 7 21:29:06.522560400 (11h49m28.266617269s) [upload=12233]: Got next segment. Inline: true Aug 7 21:29:06.522566995 (11h49m28.266623844s) [upload=12233]: Waiting for error group managing segments... Aug 7 21:29:06.522615251 (11h49m28.266672101s) [upload=12233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:06.522621757 (11h49m28.266678608s) [upload=12233]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:06.522630028 (11h49m28.266686879s) [upload=12233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:36.250046349 (11h49m57.994103211s) [upload=12233]: Done uploading segments... Aug 7 21:29:36.250088941 (11h49m57.994145808s) [upload=12233]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:03.947527959 (11h50m25.691584822s) [upload=12249]: Starting upload Aug 7 21:30:03.947767439 (11h50m25.691824311s) [upload=12249]: Done starting upload Aug 7 21:30:03.959515214 (11h50m25.703572066s) [upload=12249]: Uploading segments... Aug 7 21:30:03.959659046 (11h50m25.703715962s) [upload=12249]: Got next segment. Inline: true Aug 7 21:30:03.959668426 (11h50m25.703725302s) [upload=12249]: Waiting for error group managing segments... Aug 7 21:30:03.959783210 (11h50m25.703840082s) [upload=12249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:03.959790677 (11h50m25.703847531s) [upload=12249]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:03.959795320 (11h50m25.703852169s) [upload=12249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:33.655073335 (11h50m55.399130195s) [upload=12249]: Done uploading segments... Aug 7 21:30:33.655096560 (11h50m55.399153412s) [upload=12249]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:38.130475962 (11h50m59.874532829s) [upload=12256]: Starting upload Aug 7 21:30:38.130573915 (11h50m59.874630787s) [upload=12256]: Done starting upload Aug 7 21:30:38.130665890 (11h50m59.874722739s) [upload=12256]: Uploading segments... Aug 7 21:30:38.131017420 (11h50m59.875074302s) [upload=12256]: Got next segment. Inline: true Aug 7 21:30:38.131029217 (11h50m59.87508609s) [upload=12256]: Waiting for error group managing segments... Aug 7 21:30:38.131118811 (11h50m59.875175683s) [upload=12256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:38.131125270 (11h50m59.875182135s) [upload=12256]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:38.131131701 (11h50m59.875188569s) [upload=12256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:07.857077892 (11h51m29.601134744s) [upload=12256]: Done uploading segments... Aug 7 21:31:07.857100740 (11h51m29.601157585s) [upload=12256]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:11.159434284 (11h51m32.903491145s) [upload=12264]: Starting upload Aug 7 21:31:11.159626488 (11h51m32.903683349s) [upload=12264]: Done starting upload Aug 7 21:31:11.159711096 (11h51m32.903767956s) [upload=12264]: Uploading segments... Aug 7 21:31:11.159755148 (11h51m32.903812017s) [upload=12264]: Got next segment. Inline: true Aug 7 21:31:11.159760622 (11h51m32.903817485s) [upload=12264]: Waiting for error group managing segments... Aug 7 21:31:11.159814091 (11h51m32.903870951s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:11.159819842 (11h51m32.903876703s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:11.159823678 (11h51m32.903880539s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:40.887250927 (11h52m2.631307774s) [upload=12264]: Done uploading segments... Aug 7 21:31:40.887302452 (11h52m2.631359314s) [upload=12264]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:45.153828960 (11h52m6.89788582s) [upload=12272]: Starting upload Aug 7 21:31:45.153940931 (11h52m6.897997802s) [upload=12272]: Done starting upload Aug 7 21:31:45.154020734 (11h52m6.898077597s) [upload=12272]: Uploading segments... Aug 7 21:31:45.154082030 (11h52m6.898138908s) [upload=12272]: Got next segment. Inline: true Aug 7 21:31:45.154088359 (11h52m6.898145223s) [upload=12272]: Waiting for error group managing segments... Aug 7 21:31:45.154142898 (11h52m6.898199759s) [upload=12272]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:45.154150494 (11h52m6.898207362s) [upload=12272]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:45.154155049 (11h52m6.898211914s) [upload=12272]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:32:14.861389353 (11h52m36.605446232s) [upload=12272]: Done uploading segments... Aug 7 21:32:14.861441235 (11h52m36.605498102s) [upload=12272]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001111442s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 3 out of 60 is failed (threshold=3) [...........................................XX..............X] duration: 2m0.000747048s remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 21:32:11 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 2m0.000770622s edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.................................X..XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001458545s FAIL (2025-08-07 21:32:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001484188s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001151794s FAIL (2025-08-07 21:32:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001178846s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [..................................................XXXXXXXXXX] duration: 30.000634211s FAIL (2025-08-07 21:31:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== ing segments... Aug 7 21:19:10.410002320 (11h39m32.154059166s) [upload=12083]: Upload finished. err: context deadline exceeded Aug 7 21:19:47.988992569 (11h40m9.733049439s) [upload=12096]: Starting upload Aug 7 21:19:47.989079375 (11h40m9.733136239s) [upload=12096]: Done starting upload Aug 7 21:19:47.989212489 (11h40m9.73326935s) [upload=12096]: Uploading segments... Aug 7 21:19:47.989504628 (11h40m9.733561488s) [upload=12096]: Got next segment. Inline: false Aug 7 21:19:47.989543207 (11h40m9.733600067s) [upload=12096]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:19:47.989548063 (11h40m9.733604923s) [upload=12096]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:20:13.796948087 (11h40m35.541004946s) [seg_pos={0 0}, upload=12096]: Begin upload segment... Aug 7 21:20:13.796997392 (11h40m35.541054239s) [seg_pos={0 0}, upload=12096]: Done begin upload segment. Aug 7 21:20:13.797043921 (11h40m35.541100788s) [noise=true , address=77.90.31.18:4805 , piece_id=HKKJRNDDMITSXEXDDMZ37ZCG5OFNG72PAEDN7TNWDHXZQYBSWIXA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.797051683 (11h40m35.541108546s) [noise=true , address=104.36.82.15:4257 , piece_id=46HGSNBRDPF3JZMTPWY3OPAGK3RXBGTZWXDDNWZQERV6QAYWJ3TA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.797383845 (11h40m35.541440712s) [noise=true , address=77.90.31.17:4772 , piece_id=EMV6OLIN7YSBQQN4SC4VJGWQGFKLPBQVVJZTTKUE5ELQDXMXPSSQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.797528063 (11h40m35.541584979s) [noise=true , address=77.90.31.18:4807 , piece_id=O3SDUB6BQB6CPYN4X7Z7GHUK5Y663IOBGE4S4MB5UBB46HCQMJLA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.797616630 (11h40m35.541673477s) [noise=true , address=45.91.116.4:4937 , piece_id=PGZP47Q6KAXPGOPJ5JAAKADDBRWMAIBX5GWL6JGHFJUU3AMOPKTQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.797841016 (11h40m35.541897882s) [noise=true , address=45.91.116.10:4082 , piece_id=3AL7IMQKZU26B34QQEK43SAXSWT6W7M4TF4ITTYF774U2LED2LRQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.797890836 (11h40m35.541947696s) [noise=true , address=45.91.116.6:4973 , piece_id=UKM3AZMNW67WQ7WMFP5Z55NG4JBZBFLDVODUHIISQYZ27HUVHP6Q, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798097154 (11h40m35.542154017s) [noise=true , address=45.91.116.8:4047 , piece_id=RCBP5UBGZ5C6YEQQQ3SD3GUU26LJTFX3GOL4XZOKEFZOXB4WDRSA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798331623 (11h40m35.542388488s) [noise=true , address=104.36.82.7:4912 , piece_id=2XMUVIQB6QCEDOJGJR7CY6FW6PALYZMQRA7KNPSQQB4IAYHPQZRA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798662439 (11h40m35.542719308s) [noise=true , address=77.90.31.13:4627 , piece_id=TOKP44CMWL4OXYLWOFHL7WFKSN2C57WGHWBPZMQDVZ4GRDKLCBRA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798759384 (11h40m35.54281625s) [noise=true , address=77.90.31.10:4519 , piece_id=FDV33ZMCVYJOMT4YP7KIRA3MBB7OUNJHAPITYYLMXJW6DZWKGGCQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.798971326 (11h40m35.54302818s) [noise=true , address=45.91.116.12:4225 , piece_id=EHBNQPDYM7HB4OGRJGU6QZ3CTM5YWHUEV2SSD5LKUZI6LGG6X4GA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.799201622 (11h40m35.543258483s) [noise=true , address=104.36.82.2:4731 , piece_id=DKWMN2CLFT3XVYTGEF3HJYZXPME5TE37I45TD5ABWVRQMWE3X6YA, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.799847282 (11h40m35.543904151s) [noise=true , address=77.90.31.13:4626 , piece_id=VNGCTMK7ET2RCVPEDCDXZHYKARZIIGUU6CPVT6LKA7HCLWJFJ7AQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.800536275 (11h40m35.544593146s) [noise=true , address=45.91.116.5:4297 , piece_id=77LOHOMR6FRLTTYGTAVC4SHMSDSDWGNITJMIIKDIA7ADS2272TAQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:13.800663553 (11h40m35.544720423s) [noise=true , address=45.91.116.12:4226 , piece_id=47PAAFR7VRN7H7MBS7Z7KRONLGN6Q2JF5CBYT6ASUIANYBUSYNKQ, seg_pos={0 0}, upload=12096]: Uploading piece... Aug 7 21:20:14.404668625 (11h40m36.148725475s) [upload=12096]: Next returned nil segment Aug 7 21:20:14.404685815 (11h40m36.148742676s) [upload=12096]: Waiting for error group managing segments... Aug 7 21:20:14.561430060 (11h40m36.305486911s) [noise=true , address=45.91.116.12:4225 , piece_id=EHBNQPDYM7HB4OGRJGU6QZ3CTM5YWHUEV2SSD5LKUZI6LGG6X4GA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.583124069 (11h40m36.327180919s) [noise=true , address=104.36.82.15:4257 , piece_id=46HGSNBRDPF3JZMTPWY3OPAGK3RXBGTZWXDDNWZQERV6QAYWJ3TA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.622563656 (11h40m36.366620519s) [noise=true , address=45.91.116.8:4047 , piece_id=RCBP5UBGZ5C6YEQQQ3SD3GUU26LJTFX3GOL4XZOKEFZOXB4WDRSA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.648162660 (11h40m36.392219508s) [noise=true , address=77.90.31.17:4772 , piece_id=EMV6OLIN7YSBQQN4SC4VJGWQGFKLPBQVVJZTTKUE5ELQDXMXPSSQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.649278924 (11h40m36.393335781s) [noise=true , address=104.36.82.7:4912 , piece_id=2XMUVIQB6QCEDOJGJR7CY6FW6PALYZMQRA7KNPSQQB4IAYHPQZRA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.657067139 (11h40m36.401123988s) [noise=true , address=77.90.31.18:4807 , piece_id=O3SDUB6BQB6CPYN4X7Z7GHUK5Y663IOBGE4S4MB5UBB46HCQMJLA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.660665476 (11h40m36.404722337s) [noise=true , address=77.90.31.13:4627 , piece_id=TOKP44CMWL4OXYLWOFHL7WFKSN2C57WGHWBPZMQDVZ4GRDKLCBRA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.692922160 (11h40m36.43697902s) [noise=true , address=104.36.82.2:4731 , piece_id=DKWMN2CLFT3XVYTGEF3HJYZXPME5TE37I45TD5ABWVRQMWE3X6YA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.693482368 (11h40m36.437539243s) [noise=true , address=45.91.116.12:4226 , piece_id=47PAAFR7VRN7H7MBS7Z7KRONLGN6Q2JF5CBYT6ASUIANYBUSYNKQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.722004664 (11h40m36.466061524s) [noise=true , address=45.91.116.5:4297 , piece_id=77LOHOMR6FRLTTYGTAVC4SHMSDSDWGNITJMIIKDIA7ADS2272TAQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.747287536 (11h40m36.491344386s) [noise=true , address=45.91.116.6:4973 , piece_id=UKM3AZMNW67WQ7WMFP5Z55NG4JBZBFLDVODUHIISQYZ27HUVHP6Q, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.751701687 (11h40m36.495758555s) [noise=true , address=77.90.31.18:4805 , piece_id=HKKJRNDDMITSXEXDDMZ37ZCG5OFNG72PAEDN7TNWDHXZQYBSWIXA, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.798490539 (11h40m36.542547386s) [noise=true , address=77.90.31.10:4519 , piece_id=FDV33ZMCVYJOMT4YP7KIRA3MBB7OUNJHAPITYYLMXJW6DZWKGGCQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: <nil> Aug 7 21:20:14.798534450 (11h40m36.54259131s) [seg_pos={0 0}, upload=12096]: Segment reached optimal threshold of 13 pieces. Aug 7 21:20:14.798673827 (11h40m36.542730687s) [noise=true , address=45.91.116.10:4082 , piece_id=3AL7IMQKZU26B34QQEK43SAXSWT6W7M4TF4ITTYF774U2LED2LRQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798750267 (11h40m36.542807123s) [noise=true , address=77.90.31.13:4626 , piece_id=VNGCTMK7ET2RCVPEDCDXZHYKARZIIGUU6CPVT6LKA7HCLWJFJ7AQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798798579 (11h40m36.542855426s) [noise=true , address=45.91.116.4:4937 , piece_id=PGZP47Q6KAXPGOPJ5JAAKADDBRWMAIBX5GWL6JGHFJUU3AMOPKTQ, seg_pos={0 0}, upload=12096]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Aug 7 21:20:14.798820360 (11h40m36.54287721s) [seg_pos={0 0}, upload=12096]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 21:20:14.798910105 (11h40m36.542966966s) [upload=12096]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:20:14.798915686 (11h40m36.542972547s) [upload=12096]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:24.464709964 (11h51m46.208766834s) [upload=12096]: Done uploading segments... Aug 7 21:31:24.464745783 (11h51m46.208802653s) [upload=12096]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:24.881904902 (11h51m46.625961766s) [upload=12268]: Starting upload Aug 7 21:31:24.882094151 (11h51m46.626151016s) [upload=12268]: Done starting upload Aug 7 21:31:24.882164223 (11h51m46.626221082s) [upload=12268]: Uploading segments... Aug 7 21:31:24.882475408 (11h51m46.626532277s) [upload=12268]: Got next segment. Inline: false Aug 7 21:31:24.882552261 (11h51m46.626609127s) [upload=12268]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:24.882560165 (11h51m46.626617032s) [upload=12268]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:31:54.614102730 (11h52m16.358159588s) [upload=12268]: Next segment err: context deadline exceeded Aug 7 21:31:54.614205863 (11h52m16.358262714s) [upload=12268]: Done uploading segments... Aug 7 21:31:54.614225924 (11h52m16.358282773s) [upload=12268]: Upload finished. err: context deadline exceeded Duration: 30.000652393s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X......................X.......................] duration: 3.541562432s FAIL (2025-08-07 21:31:54 UTC) orbiter warning: operation: DownloadObject ======ERROR====== uplink: object not found ("orbiter_us_0/4M") ======LOGS====== oad finished. err: <nil> Aug 7 21:14:53.365419244 (11h35m15.109476095s) [upload=12029]: Starting upload Aug 7 21:14:53.365483423 (11h35m15.109540274s) [upload=12029]: Done starting upload Aug 7 21:14:53.365847548 (11h35m15.109904403s) [upload=12029]: Uploading segments... Aug 7 21:14:53.365943593 (11h35m15.110000458s) [upload=12029]: Got next segment. Inline: false Aug 7 21:14:53.366001458 (11h35m15.110058308s) [upload=12029]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:14:53.366007419 (11h35m15.110064269s) [upload=12029]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 21:14:53.565386761 (11h35m15.309443632s) [seg_pos={0 0}, upload=12029]: Begin upload segment... Aug 7 21:14:53.565453585 (11h35m15.309510436s) [seg_pos={0 0}, upload=12029]: Done begin upload segment. Aug 7 21:14:53.565481091 (11h35m15.309537957s) [noise=true , address=79.127.205.234:4602 , piece_id=MX35WEO2ET72CJP2BKTK7WEPQBARCOE27M7P3H7OZXPO4KCWHJKA, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.565521443 (11h35m15.309578305s) [noise=true , address=104.36.82.23:4468 , piece_id=FUQL5RAGCLECSKKYSNEP3ZNUIEKCCHG2CLWBHWTKL6O2QT23UF7A, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.566289759 (11h35m15.310346619s) [noise=true , address=104.36.82.59:4311 , piece_id=YW3HCFGQ7V5GSOEASEJ75A7RNT5URMVEW7RNPIHKYTJOPTXLRCZQ, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.566595379 (11h35m15.31065223s) [noise=true , address=104.36.82.41:4793 , piece_id=UQRKUBTVEX357CCO2LG3J6HAEZYTD4AH4SJYDMPI7JV5VEHWKVLQ, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.566764972 (11h35m15.310821839s) [noise=true , address=104.36.82.5:4859 , piece_id=KNIFV22KO2KVKCYTXUKCSZ774GMKDEYLY2AZQBUOFCGTVE2GNGKQ, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.566955689 (11h35m15.31101255s) [noise=true , address=45.91.116.4:4944 , piece_id=H5M5J52YPWDBF27RKRY6TZE4DRFOTKBFQKIIJJFNJCYF4QZ2WO2Q, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.567166790 (11h35m15.311223672s) [noise=true , address=79.127.205.233:4646 , piece_id=KXPQQXRFFQBH7DQKOKURS4EN47YIOZWAM3N7PCIA4CYA4FVJ64SQ, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.567257141 (11h35m15.311314011s) [noise=true , address=104.36.82.53:4143 , piece_id=QHDKVDXS56LDT53FKQQ57KUFIY4TYO6W3OGOQYNBKLFZ6J7URK3Q, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.567695963 (11h35m15.311752829s) [noise=true , address=104.36.82.56:4229 , piece_id=J3TJ6C45D647QCYH2LP4NEZDFCW22JUPNZAFGVVEW26FMXDEUBQA, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.567729634 (11h35m15.311786511s) [noise=true , address=104.36.82.37:4686 , piece_id=EU5QDHW44OCPNEI3355XHD5VEVUCEFTO2LEDTH626KWZFMVIF3WA, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.568042033 (11h35m15.312098893s) [noise=true , address=104.36.82.60:4358 , piece_id=3EAVHMKYL2CLWDZX3DFAGX7ZRPFEKG5IA5IOVXS4QK347T5VTOLA, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.568093826 (11h35m15.312150693s) [noise=true , address=104.36.82.15:4263 , piece_id=MHQ54RTCV6BFUT6STUGEKOJNM2N3OQ7EXUE7XC44HP2IORHTABEA, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.568392535 (11h35m15.312449404s) [noise=true , address=64.49.13.140:4510 , piece_id=AOLVILWMK2UVHN4ZHXK4B4XKJRSZSYXPZL4YN65TWUYQEONI55MQ, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.568543778 (11h35m15.312600643s) [noise=true , address=64.49.13.26:4956 , piece_id=WZU2GAEHVVGHYAKGI6AZ27VAUOTQ2KDAQAU2X4DAVNX2BIVZETRA, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.568718488 (11h35m15.312775348s) [noise=true , address=104.36.82.8:4975 , piece_id=OT2R76IEADI5LFR7SDCUTYD3YIBZHFZG6TQTWFUK6JOSBS4MXU2A, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.568895867 (11h35m15.31295273s) [noise=true , address=45.91.116.6:4002 , piece_id=UTJDKYUZI3F3KD5ZNSSKMYIZPZKZFPZDDJYNENEFIPIOZ3IX6Q3A, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.569229978 (11h35m15.313286841s) [noise=true , address=193.149.164.143:4699 , piece_id=3QIF7NXRJXYHITN3GQ6DEAAVQ5FNL3F7PPUXKJBR72BHIUSGHUVQ, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:53.570109875 (11h35m15.314166736s) [noise=true , address=104.36.82.61:4380 , piece_id=DDLKPWKAAGKDTYFNRTPUIGL7ZNTGJRQJI4EXICX3Q4KTUSR3HYYA, seg_pos={0 0}, upload=12029]: Uploading piece... Aug 7 21:14:54.148674771 (11h35m15.892731622s) [upload=12029]: Next returned nil segment Aug 7 21:14:54.148691178 (11h35m15.892748028s) [upload=12029]: Waiting for error group managing segments... Aug 7 21:14:54.307599195 (11h35m16.051656048s) [noise=true , address=104.36.82.23:4468 , piece_id=FUQL5RAGCLECSKKYSNEP3ZNUIEKCCHG2CLWBHWTKL6O2QT23UF7A, seg_pos={0 0}, upload=12029]: Done uploading piece. err: <nil> Aug 7 21:14:54.317245558 (11h35m16.061302413s) [noise=true , address=104.36.82.41:4793 , piece_id=UQRKUBTVEX357CCO2LG3J6HAEZYTD4AH4SJYDMPI7JV5VEHWKVLQ, seg_pos={0 0}, upload=12029]: Done uploading piece. err: <nil> Aug 7 21:14:54.327537824 (11h35m16.071594686s) [noise=true , address=104.36.82.60:4358 , piece_id=3EAVHMKYL2CLWDZX3DFAGX7ZRPFEKG5IA5IOVXS4QK347T5VTOLA, seg_pos={0 0}, upload=12029]: Done uploading piece. err: <nil> Aug 7 21:14:54.335711829 (11h35m16.079768688s) [noise=true , address=104.36.82.53:4143 , piece_id=QHDKVDXS56LDT53FKQQ57KUFIY4TYO6W3OGOQYNBKLFZ6J7URK3Q, seg_pos={0 0}, upload=12029]: Done uploading piece. err: <nil> Aug 7 21:14:54.340624043 (11h35m16.084680896s) [noise=true , address=104.36.82.56:4229 , piece_id=J3TJ6C45D647QCYH2LP4NEZDFCW22JUPNZAFGVVEW26FMXDEUBQA, seg_pos={0 0}, upload=12029]: Done uploading piece. err: <nil> Aug 7 21:14:54.347786380 (11h35m16.091843234s) [noise=true , address=104.36.82.8:4975 , piece_id=OT2R76IEADI5LFR7SDCUTYD3YIBZHFZG6TQTWFUK6JOSBS4MXU2A, seg_pos={0 0}, upload=12029]: Done uploading piece. err: <nil> Aug 7 21:14:54.360886940 (11h35m16.104943808s) [noise=true , address=45.91.116.6:4002 , piece_id=UTJDKYUZI3F3KD5ZNSSKMYIZPZKZFPZDDJYNENEFIPIOZ3IX6Q3A, seg_pos={0 0}, upload=12029]: Done uploading piece. err: <nil> Aug 7 21:14:54.382665858 (11h35m16.126722724s) [noise=true , address=104.36.82.37:4686 , piece_id=EU5QDHW44OCPNEI3355XHD5VEVUCEFTO2LEDTH626KWZFMVIF3WA, seg_pos={0 0}, upload=12029]: Done uploading piece. err: <nil> Aug 7 21:14:54.396988326 (11h35m16.141045179s) [noise=true , address=45.91.116.4:4944 , piece_id=H5M5J52YPWDBF27RKRY6TZE4DRFOTKBFQKIIJJFNJCYF4QZ2WO2Q, seg_pos={0 0}, upload=12029]: Done uploading piece. err: <nil> Aug 7 21:14:54.468138008 (11h35m16.212194859s) [noise=true , address=104.36.82.15:4263 , piece_id=MHQ54RTCV6BFUT6STUGEKOJNM2N3OQ7EXUE7XC44HP2IORHTABEA, seg_pos={0 0}, upload=12029]: Done uploading piece. err: <nil> Aug 7 21:14:54.474865865 (11h35m16.218922761s) [noise=true , address=104.36.82.5:4859 , piece_id=KNIFV22KO2KVKCYTXUKCSZ774GMKDEYLY2AZQBUOFCGTVE2GNGKQ, seg_pos={0 0}, upload=12029]: Done uploading piece. err: <nil> Aug 7 21:14:54.664449958 (11h35m16.408506808s) [noise=true , address=104.36.82.59:4311 , piece_id=YW3HCFGQ7V5GSOEASEJ75A7RNT5URMVEW7RNPIHKYTJOPTXLRCZQ, seg_pos={0 0}, upload=12029]: Done uploading piece. err: <nil> Aug 7 21:14:54.806820897 (11h35m16.550877751s) [noise=true , address=193.149.164.143:4699 , piece_id=3QIF7NXRJXYHITN3GQ6DEAAVQ5FNL3F7PPUXKJBR72BHIUSGHUVQ, seg_pos={0 0}, upload=12029]: Done uploading piece. err: <nil> Aug 7 21:14:54.826156011 (11h35m16.570212865s) [noise=true , address=64.49.13.140:4510 , piece_id=AOLVILWMK2UVHN4ZHXK4B4XKJRSZSYXPZL4YN65TWUYQEONI55MQ, seg_pos={0 0}, upload=12029]: Done uploading piece. err: <nil> Aug 7 21:14:54.826199084 (11h35m16.570255944s) [seg_pos={0 0}, upload=12029]: Segment reached optimal threshold of 14 pieces. Aug 7 21:14:54.826382478 (11h35m16.570439338s) [noise=true , address=79.127.205.234:4602 , piece_id=MX35WEO2ET72CJP2BKTK7WEPQBARCOE27M7P3H7OZXPO4KCWHJKA, seg_pos={0 0}, upload=12029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12XeaD9pExNha3FaRaCyyfTHBeQMrmP19MHR135yT8a7Uo4eUyZ): protocol: expected piece hash; context canceled; EOF Aug 7 21:14:54.826407613 (11h35m16.570464464s) [noise=true , address=64.49.13.26:4956 , piece_id=WZU2GAEHVVGHYAKGI6AZ27VAUOTQ2KDAQAU2X4DAVNX2BIVZETRA, seg_pos={0 0}, upload=12029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12TYHhjpfyBKkenSwviyQCUZpSBbnaFCGBFRkM4fyKLDuxKx4Ju): protocol: expected piece hash; context canceled; EOF Aug 7 21:14:54.826504181 (11h35m16.570561027s) [noise=true , address=79.127.205.233:4646 , piece_id=KXPQQXRFFQBH7DQKOKURS4EN47YIOZWAM3N7PCIA4CYA4FVJ64SQ, seg_pos={0 0}, upload=12029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fnBRPbkdAGVB8exwLJBBtpf5bpAN8Dv86mgGf3YVMEUfX4Hdc): protocol: expected piece hash; context canceled; EOF Aug 7 21:14:54.826519435 (11h35m16.570576285s) [noise=true , address=104.36.82.61:4380 , piece_id=DDLKPWKAAGKDTYFNRTPUIGL7ZNTGJRQJI4EXICX3Q4KTUSR3HYYA, seg_pos={0 0}, upload=12029]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1MqrQPmbGWnPhLYSQKxX1eKASgv9SdUM2p4WXfQYBcENHh9VPG): protocol: expected piece hash; context canceled; EOF Aug 7 21:14:54.826544440 (11h35m16.57060129s) [seg_pos={0 0}, upload=12029]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 21:14:54.826652867 (11h35m16.570709735s) [upload=12029]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 21:14:54.826659817 (11h35m16.570716781s) [upload=12029]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:14:54.989374282 (11h35m16.733431131s) [upload=12029]: Done uploading segments... Aug 7 21:14:54.989392341 (11h35m16.73344919s) [upload=12029]: Upload finished. err: <nil> Duration: 3.541594568s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.13917544s FAIL (2025-08-07 21:31:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== loading segments... Aug 7 21:21:33.493714811 (11h41m55.237771657s) [upload=12125]: Upload finished. err: <nil> Aug 7 21:22:47.998800022 (11h43m9.742856869s) [upload=12141]: Starting upload Aug 7 21:22:47.998849424 (11h43m9.742906283s) [upload=12141]: Done starting upload Aug 7 21:22:47.998919383 (11h43m9.742976233s) [upload=12141]: Uploading segments... Aug 7 21:22:47.999001069 (11h43m9.743057938s) [upload=12141]: Got next segment. Inline: true Aug 7 21:22:47.999007866 (11h43m9.743064718s) [upload=12141]: Waiting for error group managing segments... Aug 7 21:22:47.999066684 (11h43m9.743123533s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:22:47.999073371 (11h43m9.743130221s) [upload=12141]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:22:47.999077096 (11h43m9.743133947s) [upload=12141]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:22:48.164602559 (11h43m9.90865943s) [upload=12141]: Done uploading segments... Aug 7 21:22:48.164642998 (11h43m9.908699853s) [upload=12141]: Upload finished. err: <nil> Aug 7 21:23:35.990065388 (11h43m57.734122239s) [upload=12154]: Starting upload Aug 7 21:23:35.990143300 (11h43m57.734200146s) [upload=12154]: Done starting upload Aug 7 21:23:35.990217262 (11h43m57.734274108s) [upload=12154]: Uploading segments... Aug 7 21:23:35.990277036 (11h43m57.734333925s) [upload=12154]: Got next segment. Inline: true Aug 7 21:23:35.990283708 (11h43m57.734340569s) [upload=12154]: Waiting for error group managing segments... Aug 7 21:23:35.990337388 (11h43m57.734394253s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:23:35.990343849 (11h43m57.734400714s) [upload=12154]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:23:35.990347914 (11h43m57.734405001s) [upload=12154]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:23:36.150463692 (11h43m57.894520544s) [upload=12154]: Done uploading segments... Aug 7 21:23:36.150483857 (11h43m57.894540707s) [upload=12154]: Upload finished. err: <nil> Aug 7 21:24:13.830997960 (11h44m35.57505482s) [upload=12160]: Starting upload Aug 7 21:24:13.831049433 (11h44m35.575106295s) [upload=12160]: Done starting upload Aug 7 21:24:13.831151824 (11h44m35.575208678s) [upload=12160]: Uploading segments... Aug 7 21:24:13.831192510 (11h44m35.575249377s) [upload=12160]: Got next segment. Inline: true Aug 7 21:24:13.831197614 (11h44m35.575254475s) [upload=12160]: Waiting for error group managing segments... Aug 7 21:24:13.831253497 (11h44m35.57531036s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:13.831259191 (11h44m35.575316046s) [upload=12160]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:13.831262688 (11h44m35.575319551s) [upload=12160]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:13.982951222 (11h44m35.727008092s) [upload=12160]: Done uploading segments... Aug 7 21:24:13.982975878 (11h44m35.727032727s) [upload=12160]: Upload finished. err: <nil> Aug 7 21:24:48.198960455 (11h45m9.943017328s) [upload=12172]: Starting upload Aug 7 21:24:48.199050501 (11h45m9.943107348s) [upload=12172]: Done starting upload Aug 7 21:24:48.199166478 (11h45m9.943223341s) [upload=12172]: Uploading segments... Aug 7 21:24:48.199284289 (11h45m9.943341159s) [upload=12172]: Got next segment. Inline: true Aug 7 21:24:48.199309162 (11h45m9.943366036s) [upload=12172]: Waiting for error group managing segments... Aug 7 21:24:48.199360469 (11h45m9.943417336s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:48.199382302 (11h45m9.943439163s) [upload=12172]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:48.199386864 (11h45m9.943443734s) [upload=12172]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:48.353366598 (11h45m10.09742347s) [upload=12172]: Done uploading segments... Aug 7 21:24:48.353410465 (11h45m10.097467327s) [upload=12172]: Upload finished. err: <nil> Aug 7 21:25:56.601363668 (11h46m18.345420518s) [upload=12190]: Starting upload Aug 7 21:25:56.601416202 (11h46m18.345473063s) [upload=12190]: Done starting upload Aug 7 21:25:56.601478987 (11h46m18.345535837s) [upload=12190]: Uploading segments... Aug 7 21:25:56.601519567 (11h46m18.345576436s) [upload=12190]: Got next segment. Inline: true Aug 7 21:25:56.601524613 (11h46m18.345581463s) [upload=12190]: Waiting for error group managing segments... Aug 7 21:25:56.601564530 (11h46m18.345621381s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:56.601571562 (11h46m18.345628412s) [upload=12190]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:56.601574952 (11h46m18.345631806s) [upload=12190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:57.289489478 (11h46m19.033546339s) [upload=12190]: Done uploading segments... Aug 7 21:25:57.289514011 (11h46m19.033570861s) [upload=12190]: Upload finished. err: <nil> Aug 7 21:26:32.814119549 (11h46m54.558176399s) [upload=12198]: Starting upload Aug 7 21:26:32.814178314 (11h46m54.558235164s) [upload=12198]: Done starting upload Aug 7 21:26:32.814230962 (11h46m54.558287814s) [upload=12198]: Uploading segments... Aug 7 21:26:32.814314990 (11h46m54.558371858s) [upload=12198]: Got next segment. Inline: true Aug 7 21:26:32.814321493 (11h46m54.558378353s) [upload=12198]: Waiting for error group managing segments... Aug 7 21:26:32.814387101 (11h46m54.558443951s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:32.814393685 (11h46m54.558450545s) [upload=12198]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:32.814397318 (11h46m54.558454179s) [upload=12198]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:32.972796167 (11h46m54.716853027s) [upload=12198]: Done uploading segments... Aug 7 21:26:32.972815783 (11h46m54.716872632s) [upload=12198]: Upload finished. err: <nil> Aug 7 21:27:41.280232963 (11h48m3.024289826s) [upload=12214]: Starting upload Aug 7 21:27:41.280291948 (11h48m3.024348811s) [upload=12214]: Done starting upload Aug 7 21:27:41.280410716 (11h48m3.024467565s) [upload=12214]: Uploading segments... Aug 7 21:27:41.280478626 (11h48m3.024535487s) [upload=12214]: Got next segment. Inline: true Aug 7 21:27:41.280485326 (11h48m3.024542193s) [upload=12214]: Waiting for error group managing segments... Aug 7 21:27:41.280533304 (11h48m3.024590156s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:41.280538628 (11h48m3.024595488s) [upload=12214]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:41.280542274 (11h48m3.02459914s) [upload=12214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:41.433978896 (11h48m3.178035773s) [upload=12214]: Done uploading segments... Aug 7 21:27:41.434003733 (11h48m3.178060594s) [upload=12214]: Upload finished. err: <nil> Aug 7 21:28:53.851663561 (11h49m15.595720412s) [upload=12231]: Starting upload Aug 7 21:28:53.851736366 (11h49m15.595793226s) [upload=12231]: Done starting upload Aug 7 21:28:53.851804427 (11h49m15.595861277s) [upload=12231]: Uploading segments... Aug 7 21:28:53.851851609 (11h49m15.595908471s) [upload=12231]: Got next segment. Inline: true Aug 7 21:28:53.851857251 (11h49m15.595914111s) [upload=12231]: Waiting for error group managing segments... Aug 7 21:28:53.851899816 (11h49m15.595956668s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:53.851905032 (11h49m15.595961892s) [upload=12231]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:53.851908856 (11h49m15.595965718s) [upload=12231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:28:54.191384948 (11h49m15.935441808s) [upload=12231]: Done uploading segments... Aug 7 21:28:54.191405192 (11h49m15.935462045s) [upload=12231]: Upload finished. err: <nil> Aug 7 21:29:19.999054333 (11h49m41.74311118s) [upload=12237]: Starting upload Aug 7 21:29:19.999128859 (11h49m41.743185722s) [upload=12237]: Done starting upload Aug 7 21:29:19.999188080 (11h49m41.743244931s) [upload=12237]: Uploading segments... Aug 7 21:29:19.999327277 (11h49m41.74338415s) [upload=12237]: Got next segment. Inline: true Aug 7 21:29:19.999335397 (11h49m41.743392265s) [upload=12237]: Waiting for error group managing segments... Aug 7 21:29:19.999418844 (11h49m41.743475691s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:19.999426848 (11h49m41.743483712s) [upload=12237]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:19.999430264 (11h49m41.743487128s) [upload=12237]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:21.165098561 (11h49m42.909155421s) [upload=12237]: Done uploading segments... Aug 7 21:29:21.165118654 (11h49m42.909175505s) [upload=12237]: Upload finished. err: <nil> Aug 7 21:29:59.841735510 (11h50m21.585792373s) [upload=12245]: Starting upload Aug 7 21:29:59.841811008 (11h50m21.585867874s) [upload=12245]: Done starting upload Aug 7 21:29:59.841889062 (11h50m21.585945909s) [upload=12245]: Uploading segments... Aug 7 21:29:59.841961319 (11h50m21.58601819s) [upload=12245]: Got next segment. Inline: true Aug 7 21:29:59.841966490 (11h50m21.586023336s) [upload=12245]: Waiting for error group managing segments... Aug 7 21:29:59.842009456 (11h50m21.586066303s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:59.842013678 (11h50m21.586070524s) [upload=12245]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:59.842016402 (11h50m21.586073249s) [upload=12245]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:59.937966407 (11h50m21.682023257s) [upload=12245]: Done uploading segments... Aug 7 21:29:59.937989326 (11h50m21.682046187s) [upload=12245]: Upload finished. err: metaclient: Too Many Requests Duration: 1.139204943s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 21 out of 60 is failed (threshold=3) [.......................................XXXXXXXXXXXXXXXXXXXXX] duration: 30.001152455s FAIL (2025-08-07 21:31:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001175918s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 25 out of 60 is failed (threshold=3) [..................................XX.XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.000509361s FAIL (2025-08-07 21:31:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h44m42.687010195s) [upload=12164]: Uploading segments... Aug 7 21:24:20.943004925 (11h44m42.687061797s) [upload=12164]: Got next segment. Inline: true Aug 7 21:24:20.943010718 (11h44m42.687067582s) [upload=12164]: Waiting for error group managing segments... Aug 7 21:24:20.943057728 (11h44m42.687114586s) [upload=12164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:24:20.943063424 (11h44m42.687120287s) [upload=12164]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:24:20.943066967 (11h44m42.687123828s) [upload=12164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:24:50.669999700 (11h45m12.414056561s) [upload=12164]: Done uploading segments... Aug 7 21:24:50.670022022 (11h45m12.414078882s) [upload=12164]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:15.759998843 (11h45m37.504055689s) [upload=12180]: Starting upload Aug 7 21:25:15.760060918 (11h45m37.504117764s) [upload=12180]: Done starting upload Aug 7 21:25:15.760121258 (11h45m37.504178104s) [upload=12180]: Uploading segments... Aug 7 21:25:15.760159734 (11h45m37.504216604s) [upload=12180]: Got next segment. Inline: true Aug 7 21:25:15.760164169 (11h45m37.504221029s) [upload=12180]: Waiting for error group managing segments... Aug 7 21:25:15.760202724 (11h45m37.504259584s) [upload=12180]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:15.760207148 (11h45m37.504264007s) [upload=12180]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:15.760212108 (11h45m37.504268971s) [upload=12180]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:25:45.477193191 (11h46m7.22125006s) [upload=12180]: Done uploading segments... Aug 7 21:25:45.477222897 (11h46m7.22127976s) [upload=12180]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:25:45.889252639 (11h46m7.633309501s) [upload=12186]: Starting upload Aug 7 21:25:45.889337905 (11h46m7.633394765s) [upload=12186]: Done starting upload Aug 7 21:25:45.889442879 (11h46m7.633499729s) [upload=12186]: Uploading segments... Aug 7 21:25:45.889501484 (11h46m7.633558347s) [upload=12186]: Got next segment. Inline: true Aug 7 21:25:45.889507484 (11h46m7.633564344s) [upload=12186]: Waiting for error group managing segments... Aug 7 21:25:45.889551118 (11h46m7.633607968s) [upload=12186]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:25:45.889556440 (11h46m7.633613289s) [upload=12186]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:25:45.889559845 (11h46m7.633616705s) [upload=12186]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:26:15.619717489 (11h46m37.363774339s) [upload=12186]: Done uploading segments... Aug 7 21:26:15.619739739 (11h46m37.363796589s) [upload=12186]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:26:40.252456263 (11h47m1.996513132s) [upload=12201]: Starting upload Aug 7 21:26:40.252598168 (11h47m1.996655038s) [upload=12201]: Done starting upload Aug 7 21:26:40.252702071 (11h47m1.996758923s) [upload=12201]: Uploading segments... Aug 7 21:26:40.252782136 (11h47m1.996839003s) [upload=12201]: Got next segment. Inline: true Aug 7 21:26:40.252790092 (11h47m1.996846954s) [upload=12201]: Waiting for error group managing segments... Aug 7 21:26:40.252842463 (11h47m1.996899315s) [upload=12201]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:26:40.252849740 (11h47m1.996906592s) [upload=12201]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:26:40.252854153 (11h47m1.996911015s) [upload=12201]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:09.981768433 (11h47m31.725825298s) [upload=12201]: Done uploading segments... Aug 7 21:27:09.981793631 (11h47m31.725850483s) [upload=12201]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:27:26.837217755 (11h47m48.581274619s) [upload=12211]: Starting upload Aug 7 21:27:26.837300223 (11h47m48.581357084s) [upload=12211]: Done starting upload Aug 7 21:27:26.837385984 (11h47m48.581442831s) [upload=12211]: Uploading segments... Aug 7 21:27:26.837444443 (11h47m48.581501313s) [upload=12211]: Got next segment. Inline: true Aug 7 21:27:26.837449770 (11h47m48.581506619s) [upload=12211]: Waiting for error group managing segments... Aug 7 21:27:26.837517257 (11h47m48.581574107s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:27:26.837523154 (11h47m48.581580003s) [upload=12211]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:27:26.837527097 (11h47m48.58158395s) [upload=12211]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:27:56.564982463 (11h48m18.309039313s) [upload=12211]: Done uploading segments... Aug 7 21:27:56.565004020 (11h48m18.309060871s) [upload=12211]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:28:36.367399060 (11h48m58.11145591s) [upload=12226]: Starting upload Aug 7 21:28:36.367588554 (11h48m58.111645417s) [upload=12226]: Done starting upload Aug 7 21:28:36.367664912 (11h48m58.111721765s) [upload=12226]: Uploading segments... Aug 7 21:28:36.367723895 (11h48m58.111780757s) [upload=12226]: Got next segment. Inline: true Aug 7 21:28:36.367729331 (11h48m58.111786192s) [upload=12226]: Waiting for error group managing segments... Aug 7 21:28:36.367785894 (11h48m58.111842754s) [upload=12226]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:28:36.367791010 (11h48m58.11184787s) [upload=12226]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:28:36.367794492 (11h48m58.111851352s) [upload=12226]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:06.092890267 (11h49m27.836947129s) [upload=12226]: Done uploading segments... Aug 7 21:29:06.092932205 (11h49m27.836989051s) [upload=12226]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:29:06.522310183 (11h49m28.266367033s) [upload=12233]: Starting upload Aug 7 21:29:06.522388772 (11h49m28.266445633s) [upload=12233]: Done starting upload Aug 7 21:29:06.522479144 (11h49m28.266535995s) [upload=12233]: Uploading segments... Aug 7 21:29:06.522560400 (11h49m28.266617269s) [upload=12233]: Got next segment. Inline: true Aug 7 21:29:06.522566995 (11h49m28.266623844s) [upload=12233]: Waiting for error group managing segments... Aug 7 21:29:06.522615251 (11h49m28.266672101s) [upload=12233]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:29:06.522621757 (11h49m28.266678608s) [upload=12233]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:29:06.522630028 (11h49m28.266686879s) [upload=12233]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:29:36.250046349 (11h49m57.994103211s) [upload=12233]: Done uploading segments... Aug 7 21:29:36.250088941 (11h49m57.994145808s) [upload=12233]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:03.947527959 (11h50m25.691584822s) [upload=12249]: Starting upload Aug 7 21:30:03.947767439 (11h50m25.691824311s) [upload=12249]: Done starting upload Aug 7 21:30:03.959515214 (11h50m25.703572066s) [upload=12249]: Uploading segments... Aug 7 21:30:03.959659046 (11h50m25.703715962s) [upload=12249]: Got next segment. Inline: true Aug 7 21:30:03.959668426 (11h50m25.703725302s) [upload=12249]: Waiting for error group managing segments... Aug 7 21:30:03.959783210 (11h50m25.703840082s) [upload=12249]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:03.959790677 (11h50m25.703847531s) [upload=12249]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:03.959795320 (11h50m25.703852169s) [upload=12249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:30:33.655073335 (11h50m55.399130195s) [upload=12249]: Done uploading segments... Aug 7 21:30:33.655096560 (11h50m55.399153412s) [upload=12249]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:30:38.130475962 (11h50m59.874532829s) [upload=12256]: Starting upload Aug 7 21:30:38.130573915 (11h50m59.874630787s) [upload=12256]: Done starting upload Aug 7 21:30:38.130665890 (11h50m59.874722739s) [upload=12256]: Uploading segments... Aug 7 21:30:38.131017420 (11h50m59.875074302s) [upload=12256]: Got next segment. Inline: true Aug 7 21:30:38.131029217 (11h50m59.87508609s) [upload=12256]: Waiting for error group managing segments... Aug 7 21:30:38.131118811 (11h50m59.875175683s) [upload=12256]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:30:38.131125270 (11h50m59.875182135s) [upload=12256]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:30:38.131131701 (11h50m59.875188569s) [upload=12256]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:07.857077892 (11h51m29.601134744s) [upload=12256]: Done uploading segments... Aug 7 21:31:07.857100740 (11h51m29.601157585s) [upload=12256]: Upload finished. err: metaclient: context deadline exceeded Aug 7 21:31:11.159434284 (11h51m32.903491145s) [upload=12264]: Starting upload Aug 7 21:31:11.159626488 (11h51m32.903683349s) [upload=12264]: Done starting upload Aug 7 21:31:11.159711096 (11h51m32.903767956s) [upload=12264]: Uploading segments... Aug 7 21:31:11.159755148 (11h51m32.903812017s) [upload=12264]: Got next segment. Inline: true Aug 7 21:31:11.159760622 (11h51m32.903817485s) [upload=12264]: Waiting for error group managing segments... Aug 7 21:31:11.159814091 (11h51m32.903870951s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:31:11.159819842 (11h51m32.903876703s) [upload=12264]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:31:11.159823678 (11h51m32.903880539s) [upload=12264]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:31:40.887250927 (11h52m2.631307774s) [upload=12264]: Done uploading segments... Aug 7 21:31:40.887302452 (11h52m2.631359314s) [upload=12264]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000535863s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..................................X..XXXXXXXXXXXXXXXXXXXXXXX] duration: 30.001354492s FAIL (2025-08-07 21:31:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001367756s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)