Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.325056231s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:13:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.325088542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.269351145s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:12:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.269382387s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.345064059s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:12:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.345110232s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.24618634s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:11:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.246210124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.277819421s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:10:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.277846582s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 3.091791611s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:10:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 3.09181861s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.228845537s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:09:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.228869714s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.277192579s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:08:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.27721737s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.254409574s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:08:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.25444036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.404626249s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:07:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.404655747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.340250631s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:07:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.340281939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.246519267s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:06:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.246547638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.367230493s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:05:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.367333576s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.328487233s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:05:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.328520463s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.208841483s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:04:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.208866695s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.308951025s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:03:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.308969947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.548340392s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:02:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.548370432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.415412564s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:02:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.415439619s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.297008857s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:02:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.29703466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.398983136s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:01:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2.399001818s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 2m43.319434305s trace: https://jaeger.dev.storj.io/trace/6008bf17bf387fc6 FAIL (2025-10-30 13:01:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.237882093 (1h17m48.939474542s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.238888034 (1h17m48.940480489s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.239647527 (1h17m48.941239984s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.240409346 (1h17m48.942001802s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241178882 (1h17m48.942771334s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.241974741 (1h17m48.9435672s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.242795572 (1h17m48.94438803s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.243586710 (1h17m48.945179163s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.246163587 (1h17m48.947756042s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.248792738 (1h17m48.950385191s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.251308838 (1h17m48.952901296s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.253691763 (1h17m48.955284222s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.258790036 (1h17m48.96038249s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.262799663 (1h17m48.964392112s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.266524529 (1h17m48.968116986s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Uploading piece... Oct 30 12:58:46.818481949 (1h17m49.520074428s) [upload=1146]: Next returned nil segment Oct 30 12:58:46.818505620 (1h17m49.52009808s) [upload=1146]: Waiting for error group managing segments... Oct 30 12:58:46.973096040 (1h17m49.674688499s) [noise=true , address=145.79.227.21:4506 , piece_id=NUHAHDKQUORUEUHR2G5XB2HM4CE5R54NZGNDXGF4AIDED4O2L6UA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976507046 (1h17m49.678099529s) [noise=true , address=145.79.227.13:4820 , piece_id=4AB6QBMV6ANYYSPEOILYRW3ZPJC7G4W6PLMOXNANXF2DHUF44KBA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.976591993 (1h17m49.678184471s) [noise=true , address=45.91.116.14:4201 , piece_id=FHYSQ4FYNH5FC7UEMMDJ53T6442D334KHH7SOH65PS26F3UDQCPA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.978856565 (1h17m49.680449024s) [noise=true , address=77.90.31.13:4658 , piece_id=I4LRU25G4U4AAP2V2UX7FVD3RGVFOEWC56NDEW7N5IQH223XZNNQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985049256 (1h17m49.686641714s) [noise=true , address=145.79.227.52:4475 , piece_id=VW3TDYK5FIBP7QL4ZKWEBMGAQINBEJILM47KWIE24DFSRT3TJ5FQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985184031 (1h17m49.686776483s) [noise=true , address=77.90.31.21:4910 , piece_id=N5YZY6GHLTUGI7VCWXILGAHYSMRDU5JEXYY7IWADDJTKK5QDZYHQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.985802999 (1h17m49.687395451s) [noise=true , address=77.90.31.12:4609 , piece_id=ALKQNK6EZHJQ4BPEJFAK2UZNTCMSHMMFT5LDR5BLJFCSRMX2E2MQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990228352 (1h17m49.691820804s) [noise=true , address=45.91.116.6:4987 , piece_id=KFM7I3A6W733EC6E3RIX4LXRLMBZ5QZDTBZIN3BKMJNCGK4GXGZA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.990387670 (1h17m49.691980124s) [noise=true , address=77.90.31.18:4815 , piece_id=4RVBI4X3OGLW355NA2EPXE4SL3V2YM3ZJPOTACIBN3BU6BHCUPIQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.992600578 (1h17m49.694193042s) [noise=true , address=77.90.31.17:4795 , piece_id=NL335G3PD24JW5QO7DQHDUVHFMNR2HYXRJ6PIFD5EZEWZGOZJVCQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.994639995 (1h17m49.696232454s) [noise=true , address=77.90.31.19:4865 , piece_id=5DHTM4N7ZKMCG3W5T236TEUOFOF6L7VLD3X6CLWX3IOUB7C6STWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:46.995631572 (1h17m49.697224027s) [noise=true , address=45.91.116.7:4018 , piece_id=KL4YPZ5WQ3TOZO73H3JAHI764BQXMY3VGAIUG7BA6KU5IG5VVRWA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.005239697 (1h17m49.706832152s) [noise=true , address=145.79.227.18:4025 , piece_id=JTRIBQDMHSCC6FHCYLYGFAAY4XWW45IOGGFXSRHKXJHSDNINJ3NA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.008899599 (1h17m49.710492069s) [noise=true , address=77.90.31.14:4684 , piece_id=NBBR4MYIWEKQCDKTS3BRPBNI5QCLCNSF3HQR7DKXJ2TJ266N7YZQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: <nil> Oct 30 12:58:47.009104449 (1h17m49.710696898s) [seg_pos={0 0}, upload=1146]: Segment reached optimal threshold of 14 pieces. Oct 30 12:58:47.009448495 (1h17m49.711040972s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12V53FpFLqgnXXEZSaUboULEwVpygzejqxEWdLgATj9MJVzacME): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009458544 (1h17m49.711051016s) [noise=true , address=45.91.116.12:4256 , piece_id=CJPPJR5LWGNM4FRAASF62TWORQXG52W4K6BW5WCSUXB4Z35J47MA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.009738945 (1h17m49.711331402s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Oct 30 12:58:47.009749269 (1h17m49.71134173s) [noise=true , address=77.90.31.11:4585 , piece_id=776TFPXATQZJLDFGDZ4BRIC3YJ7OVICO3RHOUDZZ2YPZPFQI7MAQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010098663 (1h17m49.711691122s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1iXXTKFCy3W3p3hTrHSBPFmcq7LfGnFKg4o76iFSvzFhf1vK6y): protocol: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:48776->66.23.193.175:4562: use of closed network connection Oct 30 12:58:47.010109845 (1h17m49.711702306s) [noise=true , address=66.23.193.175:4562 , piece_id=MDUC7XZVZ7D6Q32YTZZN4H2YVGD3HUPHMANR6LN6I3WCISEHG3GQ, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010145876 (1h17m49.711738336s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZedZCGumnM48qSS4jd68SuW45Q5SMqxY4UZvLrBkoWX4i6syp): protocol: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:56092->66.23.193.175:4559: use of closed network connection Oct 30 12:58:47.010159803 (1h17m49.711752263s) [noise=true , address=66.23.193.175:4559 , piece_id=YCODW42O563V3AIK7Z2PNIKNYXOVSJKHKIUH6NNIDG3UE6H4NONA, seg_pos={0 0}, upload=1146]: Long tail cancellation due to optimal threshold Oct 30 12:58:47.010206607 (1h17m49.711799055s) [seg_pos={0 0}, upload=1146]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Oct 30 12:58:47.010307415 (1h17m49.711899868s) [upload=1146]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 30 12:58:47.010315368 (1h17m49.711907823s) [upload=1146]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 30 13:01:29.130913706 (1h20m31.832506199s) [upload=1146]: Done uploading segments... Oct 30 13:01:29.130992150 (1h20m31.832584612s) [upload=1146]: Upload finished. err: metaclient: context deadline exceeded Duration: 2m43.31945843s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.669571034s FAIL (2025-10-29 19:54:35 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.669601797s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.109695682s FAIL (2025-10-29 19:53:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.109712771s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.585511361s FAIL (2025-10-29 19:52:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.585543837s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.737179421s FAIL (2025-10-29 19:52:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.737214755s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.209852773s FAIL (2025-10-29 19:51:34 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.209874813s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.722814753s FAIL (2025-10-29 19:51:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.722842601s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.831097066s FAIL (2025-10-29 19:50:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.83134161s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.628086212s FAIL (2025-10-29 19:49:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.628112356s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.707077448s FAIL (2025-10-29 19:49:10 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.707110066s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.610898676s FAIL (2025-10-29 19:47:59 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.610922638s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.635413569s FAIL (2025-10-29 19:47:05 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.635438804s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.962896053s FAIL (2025-10-29 19:46:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.962922253s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.625455451s FAIL (2025-10-29 19:45:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.625480512s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 1.659903838s FAIL (2025-10-29 19:44:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.659928701s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 1.685237516s FAIL (2025-10-29 19:43:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.685263575s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 1.674873215s FAIL (2025-10-29 19:42:50 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.674901604s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 1.672323933s FAIL (2025-10-29 19:42:25 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.672371187s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 1.62489646s FAIL (2025-10-29 19:41:15 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.624950757s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 1.695858937s FAIL (2025-10-29 19:40:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.695895045s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 1.899404392s FAIL (2025-10-29 19:40:22 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.899450943s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 1.731349541s FAIL (2025-10-29 19:39:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.731384805s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 1.563335758s FAIL (2025-10-29 19:38:20 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.563379371s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.205959476s FAIL (2025-10-29 19:37:06 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.20598601s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 1.830721709s FAIL (2025-10-29 19:36:17 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.830750471s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.60913842s FAIL (2025-10-29 19:35:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.60917075s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.088445513s FAIL (2025-10-29 19:34:18 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.08849231s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 1.684093533s FAIL (2025-10-29 19:33:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.684119476s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 1.618201079s FAIL (2025-10-29 19:32:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.618263656s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 3/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 1.638369052s FAIL (2025-10-29 19:31:51 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.638390641s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)