Orbiter History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 1.358506062s trace: https://jaeger.dev.storj.io/trace/7cb81119fbca31c8 FAIL (2025-10-30 14:53:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.358541396s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 1.488640712s trace: https://jaeger.dev.storj.io/trace/7cb81119fbca31c8 FAIL (2025-10-30 14:52:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.488676784s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 1.009304674s trace: https://jaeger.dev.storj.io/trace/7cb81119fbca31c8 FAIL (2025-10-30 14:51:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.00932197s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m50.054536193s trace: https://jaeger.dev.storj.io/trace/7cb81119fbca31c8 FAIL (2025-10-30 14:51:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1m50.054562934s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.308411221s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:41:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.308545096s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.281421485s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:41:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.281448971s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.226915052s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:40:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.226942835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.244146958s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:39:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.244187508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.532912253s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:39:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.532951003s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.305783619s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:37:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.305810522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.338243817s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:37:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.338267317s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.374725102s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:36:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.374756574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.762661358s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:36:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.762754646s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.489442672s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:35:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.489465687s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.268358598s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:34:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.26839428s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.383316924s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:34:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.38335091s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.361210541s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:33:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.361233518s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.478614481s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:33:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.478712921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.572656772s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:32:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.572789169s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.329736524s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:30:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.329766266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.304039069s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:30:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.304062818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.393397761s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:29:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.393496964s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.348886194s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:28:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.348916581s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.207932383s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:27:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.207959203s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.315403145s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:27:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.315459816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.290711993s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:26:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.29073624s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.301322354s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:25:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.3013527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.291044225s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:24:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.291085089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.314555634s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:24:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.314584733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.404846251s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:24:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.404881497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.324805069s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:23:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.324848244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.255993221s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:22:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.256020944s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.368066093s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:22:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.368093053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.456262605s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:22:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.4564051s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.25029761s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:20:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.250327757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.276429196s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:20:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.276464092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.328554621s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:19:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.328590118s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.324230156s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:19:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.324258391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.319475072s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:18:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.319498959s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.322534688s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:17:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.322563734s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.426919744s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:17:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.427026996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.35386014s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:15:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.353906137s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.355070871s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:14:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.355099288s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.325056231s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:13:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.325088542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.269351145s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:12:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.269382387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.345064059s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:12:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.345110232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.24618634s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:11:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.246210124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.277819421s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:10:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.277846582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 3.091791611s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:10:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.09181861s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.228845537s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:09:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.228869714s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)