Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 1.658043715s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:30:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.658074684s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 1.687698955s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:29:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.68773344s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 1.707938019s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:27:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.70797248s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 1.784362142s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:27:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.784409973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 1.6917201s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:26:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.691748685s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 1.65362749s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:25:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.653693919s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 1.62892543s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:24:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.628957237s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 1.686518836s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:23:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.686551463s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 1.774635011s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:22:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.774736732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 1.682219889s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:22:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.682307885s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 1.693134627s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:21:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.693174337s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 1.695553167s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:20:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.695984307s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 2.175456651s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:19:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.175542037s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 1.703786677s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:18:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.703817622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 2.113533938s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:17:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.113566046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.365069151s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:17:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.36511156s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.362809575s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:17:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.362849706s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 1.658492578s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:16:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.65851564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 2.390069175s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:16:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.390099658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 2.417735626s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:15:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.41777007s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 1.63498584s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:15:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.63516174s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 1.62568196s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:14:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.625713241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 2.26585134s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:14:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.265890569s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 1.631242774s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:14:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.631279374s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 1.783860877s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:13:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.783890619s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 2.424306474s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:13:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.424339589s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 2.121988942s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:13:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.122020542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 2.226440979s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:12:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.226474956s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 1.640116452s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:12:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.640142504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 2.367413341s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:11:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.367555474s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 1.629366067s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:11:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.629395316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 2.314779876s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:11:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.314810524s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 1.696000502s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:11:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.69606618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 2.294692513s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:10:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.2947266s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 1.847178136s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:10:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.847251145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 2.375532819s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:09:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.375594625s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 1.669653179s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:09:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.669685231s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 2.383557396s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:08:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.383601394s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 1.690565245s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:08:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.690632609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 1.654706616s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:07:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.654914154s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 2.479309198s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:07:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.479424233s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 2.329291908s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:06:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.32943253s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 1.667196644s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:06:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.66828688s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 1.687059173s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:06:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.687095147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 2.332079106s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:05:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.332112198s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.............XX............................................] duration: 2.39953809s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:05:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.399574231s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.............XX...........................................] duration: 2.295731871s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:04:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.295766706s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 2.325467567s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:04:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.325489302s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 1.622504231s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:04:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.622572071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 1.633484016s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:03:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.633522548s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)