Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.650667648s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:40:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.65070483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.112313207s trace: https://jaeger.dev.storj.io/trace/6b0c470f2b514feb FAIL (2025-11-10 02:39:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 10 02:39:05.968839802 (254h58m8.670432251s) [upload=224238]: Starting upload Nov 10 02:39:05.969083636 (254h58m8.670676092s) [upload=224238]: Done starting upload Nov 10 02:39:05.969196355 (254h58m8.670788803s) [upload=224238]: Uploading segments... Nov 10 02:39:05.969530409 (254h58m8.671122858s) [upload=224238]: Got next segment. Inline: false Nov 10 02:39:05.969595654 (254h58m8.671188108s) [upload=224238]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 02:39:05.969602003 (254h58m8.671194454s) [upload=224238]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 02:39:06.078698522 (254h58m8.780290982s) [seg_pos={0 0}, upload=224238]: Begin upload segment... Nov 10 02:39:06.078781139 (254h58m8.780373592s) [seg_pos={0 0}, upload=224238]: Done begin upload segment. Nov 10 02:39:06.078822898 (254h58m8.780415357s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.078942943 (254h58m8.780535403s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.079598720 (254h58m8.781191182s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.081030974 (254h58m8.782623433s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.082165496 (254h58m8.783757948s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.083850952 (254h58m8.785443427s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084179759 (254h58m8.78577221s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084826792 (254h58m8.786419253s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.084989073 (254h58m8.786581534s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085313041 (254h58m8.786905512s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.085732259 (254h58m8.78732472s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086385294 (254h58m8.787977747s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.086812225 (254h58m8.788404674s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087328839 (254h58m8.788921298s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.087841959 (254h58m8.789434409s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.088317270 (254h58m8.789909733s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Uploading piece... Nov 10 02:39:06.709006682 (254h58m9.410599125s) [upload=224238]: Next returned nil segment Nov 10 02:39:06.709019719 (254h58m9.41061216s) [upload=224238]: Waiting for error group managing segments... Nov 10 02:39:06.838369837 (254h58m9.539962285s) [noise=true , address=45.91.116.14:4191 , piece_id=X3ECZA4NXFGQCGXT6WZI7SSXUGGYNQENOCGX2APEMEK2ANS7FLSA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.870931240 (254h58m9.572523689s) [noise=true , address=77.90.31.15:4699 , piece_id=YRJSZZYIDBHXHOMFAICSDBFZMQK2NGGW4LOIIKBEW3HM7S6UQXFA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:06.873796889 (254h58m9.575389341s) [noise=true , address=45.91.116.4:4940 , piece_id=7RF3RHNYP3KJSOQNKSTMJ7NRHG6QTPG5OAABK32FAND7HEDAF2UQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.555449455 (254h58m10.257041908s) [noise=true , address=145.79.227.10:4403 , piece_id=23BTP4CNXEFBLUZV53KKMM37PFE4ZNTUOKYGCV3CKPLUWDDQE5DQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.695267463 (254h58m10.396859911s) [noise=true , address=145.79.227.21:4475 , piece_id=JULAJO65TAJMYDD7VAKB4H4DNXJOENRXLUYUI5GVSDI6NAGOEUHA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:07.757587810 (254h58m10.459180268s) [noise=true , address=145.79.227.13:4802 , piece_id=2ADO7VLCGEVPMHTEBBO3H2EGFHTXOZRYNNSH2CRI5OUWRT5TWSMA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.044722511 (254h58m10.746314949s) [noise=true , address=145.79.227.16:4912 , piece_id=XT7UYNE2IPFVZIORLY257GNOTYADOKOFM54BFFRMESEPZLIRBD4Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:08.066271459 (254h58m10.76786391s) [noise=true , address=145.79.227.31:4798 , piece_id=OUI5XLORCPC67HYYQRMCLE4WEEWOYYPXAC6PNV3AT2C5KKTCLI3Q, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.747851326 (254h58m12.449443774s) [noise=true , address=145.79.227.17:4970 , piece_id=ADBRFXL3RLC3NOHSPGHQXSK2NCMEBR4YOCUYR3CM7FUNZYH6X5FA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:09.816357852 (254h58m12.517950304s) [noise=true , address=145.79.227.28:4725 , piece_id=GG7KZQHC57MDCDY7GZRSHTSW5B3SRFEHL4I4PC3A6SVDZ4CXZC2A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:10.409412501 (254h58m13.11100495s) [noise=true , address=145.79.227.14:4838 , piece_id=V4PZVFJ5ZXDSTY5PORWTCXS22FGCWQI2BLYQWKD36LBKNGLET34A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:11.944972945 (254h58m14.646565393s) [noise=true , address=145.79.227.31:4799 , piece_id=EKLY7YILK3B2LFIQGDIEFTWX2WSFLY3KELF3ED3SOGK3DFQN4DCA, seg_pos={0 0}, upload=224238]: Done uploading piece. err: <nil> Nov 10 02:39:35.691440689 (254h58m38.393033148s) [noise=true , address=145.79.227.14:4837 , piece_id=44Z673V7XOXPA6J3IU53ASIJAWAANV2YKW6UUWZEUKUTGEC7DSVQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:41226->145.79.227.14:4837: use of closed network connection Nov 10 02:39:35.691554160 (254h58m38.393146628s) [noise=true , address=145.79.227.27:4690 , piece_id=OFPG3RXAWQDYNFDDERKUWWBRYV2RBFIMG4M36KSHPBH32T7YW4KQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:43546->145.79.227.27:4690: use of closed network connection Nov 10 02:39:35.691783870 (254h58m38.393376317s) [noise=true , address=145.79.227.15:4876 , piece_id=G7BKATPDPVMVUBKY6VLRWNSLHR6JCSLUYD4HY6K4KHLDWBI24C5A, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:145.79.227.15:4876): protocol: read tcp 139.59.144.22:35776->145.79.227.15:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 02:39:35.692006510 (254h58m38.393598965s) [noise=true , address=145.79.227.9:4366 , piece_id=HTBVY77NQGITF5A4RHNS3DH7R2Y6VMWWF2ZH7DNLTOUZ3DNX3IRQ, seg_pos={0 0}, upload=224238]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 139.59.144.22:52490->145.79.227.9:4366: use of closed network connection; context deadline exceeded Nov 10 02:39:35.692038241 (254h58m38.393630678s) [seg_pos={0 0}, upload=224238]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 10 02:39:35.801229231 (254h58m38.502821686s) [upload=224238]: Done uploading segments... Nov 10 02:39:35.801276269 (254h58m38.502868729s) [upload=224238]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.112356246s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 1.603044206s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:39:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.603164084s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 1.587385133s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:38:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.587418519s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 1.689639467s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:38:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.68969581s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 1.689809403s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:37:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.689861618s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 1.835695477s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:36:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.835727561s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 1.609116304s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:36:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.60921966s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 1.651953746s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:35:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.65200696s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 1.720213121s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:34:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.720245317s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 1.666211682s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:33:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.666291553s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 1.773432332s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:33:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.773487818s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 1.722650999s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:32:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.722675462s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 1.714442343s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:31:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.714515111s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 1.564951796s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:31:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.564974758s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 1.610081837s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:30:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.61010872s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 1.658043715s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:30:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.658074684s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 1.687698955s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:29:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.68773344s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 1.707938019s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:27:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.70797248s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 1.784362142s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:27:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.784409973s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 1.6917201s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:26:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.691748685s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 1.65362749s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:25:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.653693919s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 1.62892543s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:24:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.628957237s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 1.686518836s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:23:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.686551463s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 1.774635011s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:22:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.774736732s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 1.682219889s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:22:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.682307885s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 1.693134627s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:21:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.693174337s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 1.695553167s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:20:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.695984307s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 2.175456651s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:19:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.175542037s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 1.703786677s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:18:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.703817622s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 2.113533938s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:17:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.113566046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.365069151s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:17:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.36511156s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.362809575s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:17:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.362849706s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 1.658492578s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:16:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.65851564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 2.390069175s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:16:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.390099658s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 2.417735626s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:15:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.41777007s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 1.63498584s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:15:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.63516174s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 1.62568196s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:14:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.625713241s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 2.26585134s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:14:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.265890569s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 1.631242774s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:14:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.631279374s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 1.783860877s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:13:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.783890619s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 2.424306474s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:13:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.424339589s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 2.121988942s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:13:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.122020542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 2.226440979s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:12:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.226474956s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 1.640116452s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:12:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.640142504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 2.367413341s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:11:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.367555474s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 1.629366067s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:11:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.629395316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 2.314779876s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:11:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.314810524s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 1.696000502s trace: https://jaeger.dev.storj.io/trace/56c21b83858eaaf8 FAIL (2025-11-10 02:11:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.69606618s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 2.294692513s trace: https://jaeger.dev.storj.io/trace/c055a066e80622b FAIL (2025-11-10 02:10:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 10 01:29:50.180949372 (253h48m52.882541824s) [upload=223211]: Starting upload Nov 10 01:29:50.181032473 (253h48m52.882624934s) [upload=223211]: Done starting upload Nov 10 01:29:50.181156682 (253h48m52.882749125s) [upload=223211]: Uploading segments... Nov 10 01:29:50.181465875 (253h48m52.883058315s) [upload=223211]: Got next segment. Inline: false Nov 10 01:29:50.181499404 (253h48m52.883091854s) [upload=223211]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 10 01:29:50.181504566 (253h48m52.883097006s) [upload=223211]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 10 01:29:50.288289364 (253h48m52.989881813s) [seg_pos={0 0}, upload=223211]: Begin upload segment... Nov 10 01:29:50.288363675 (253h48m52.989956127s) [seg_pos={0 0}, upload=223211]: Done begin upload segment. Nov 10 01:29:50.288400256 (253h48m52.989992693s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288416601 (253h48m52.990009056s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.288921316 (253h48m52.990513768s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289398939 (253h48m52.99099142s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.289886286 (253h48m52.991478737s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290023453 (253h48m52.991615891s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290402347 (253h48m52.991994805s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290460076 (253h48m52.992052527s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.290860429 (253h48m52.992452885s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291265673 (253h48m52.992858124s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.291688246 (253h48m52.993280698s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292140757 (253h48m52.993733216s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.292895942 (253h48m52.994488396s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.293619050 (253h48m52.995211507s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294091314 (253h48m52.995683765s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.294213128 (253h48m52.995805644s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Uploading piece... Nov 10 01:29:50.898598865 (253h48m53.600191317s) [upload=223211]: Next returned nil segment Nov 10 01:29:50.898616587 (253h48m53.600209045s) [upload=223211]: Waiting for error group managing segments... Nov 10 01:29:51.030451931 (253h48m53.732044379s) [noise=true , address=77.90.31.10:4520 , piece_id=FWC3GNZIQHRM3TBWUI42BYKLCNZ3LMYZNOMAFZEGEZS2LDWMHYBA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051603630 (253h48m53.753196085s) [noise=true , address=77.90.31.15:4699 , piece_id=ZTT7EEHL2PBGXEEDZ5GOSHHCCQLSSIAMHK7AUDDFPCDS75FYQLOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.051711369 (253h48m53.75330382s) [noise=true , address=77.90.31.21:4878 , piece_id=LRYUY6I45PIRGCLW6TH5DT4NZ4BCG7FSWTQ5KNQV6YDWQVVCRPCA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.532787951 (253h48m54.234380399s) [noise=true , address=145.79.227.15:4876 , piece_id=THTJQZY5CJFDOZ64OI7RD3VP5EEOKHVT4T6MS5G3Z5AEQO2AVVBQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:51.765229559 (253h48m54.466822016s) [noise=true , address=145.79.227.26:4655 , piece_id=SYKVYBTUO2M3VZ34MHIRAMEGVGBM6OZWUYQ46C35VSZYETX2FEZQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009059018 (253h48m54.710651469s) [noise=true , address=145.79.227.6:4258 , piece_id=ADOYO4XPHEUPCBE7RR6IM4WIQXCAFSAYC7OVY56PD62Y43HDWTOA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.009456880 (253h48m54.711049319s) [noise=true , address=145.79.227.28:4726 , piece_id=53QO5GLB3GC64YZ3Z6ALE3O4KSPDNHSDE7QKWMX7SPHYWXLCDBTA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.249419947 (253h48m54.9510124s) [noise=true , address=145.79.227.13:4803 , piece_id=ZPOECQBGXDFAM5MYW5CM73XRS56R44CG4UD74SCHBQEVTCGPEBHQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:52.747418960 (253h48m55.44901141s) [noise=true , address=145.79.227.20:4438 , piece_id=X26D2AKZ2CXYBKKBTFP6TB3MWL5LSW4JKGGZN43CIU3AU5IN3JWQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.507082237 (253h48m56.208674693s) [noise=true , address=145.79.227.30:4762 , piece_id=M2V6D3VWJMURCFRK5YFNVMAO2AIOBM2GWSL6OBZJE5HUY77IJLFQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:29:53.566714417 (253h48m56.268306873s) [noise=true , address=145.79.227.32:4835 , piece_id=SQ237HBZSYOFGIJHNRBJ3W4R5YVMQ6QLAMCJRSMLMQ4E76RFVFDA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: <nil> Nov 10 01:30:19.906744987 (253h49m22.608337469s) [noise=true , address=145.79.227.10:4403 , piece_id=DKPD6QFB4VDYUTF46OQZIYDLXA6XTSWDOMZDI2O35RIWIQSYWUOQ, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12SWX9WeN8GXMsV3tUyArHw4As5vKwJRzyMxPZRMug4iAY42vh4, address:145.79.227.10:4403): protocol: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:35544->145.79.227.10:4403: use of closed network connection Nov 10 01:30:19.906915176 (253h49m22.608507655s) [noise=true , address=145.79.227.8:4331 , piece_id=AHJM74RULWU2OLNPHSARMLJWUB5L343PESDONTTWECSQB7SI4Y2A, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:145.79.227.8:4331): protocol: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:57352->145.79.227.8:4331: use of closed network connection Nov 10 01:30:19.907148249 (253h49m22.608740766s) [noise=true , address=145.79.227.14:4837 , piece_id=6UJFDJPFALMG7HYK5DBZU7VV4GU27ROOFXOLG7WVSRQE6JBZMYKA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:145.79.227.14:4837): protocol: read tcp 139.59.144.22:48446->145.79.227.14:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 10 01:30:19.907253647 (253h49m22.608846101s) [noise=true , address=145.79.227.16:4911 , piece_id=UIG36LK3PQNRTF5F47JI6UM5ITPLPPQNB3R4ORBHSLCLAST7NAPA, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:145.79.227.16:4911): protocol: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38920->145.79.227.16:4911: use of closed network connection Nov 10 01:30:19.907481275 (253h49m22.609073737s) [noise=true , address=145.79.227.21:4474 , piece_id=ITRP3SCJJ7JRC2SW54DV7CWXTVZHVJDGFSPA2VDR45CNX6UAAG3Q, seg_pos={0 0}, upload=223211]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:145.79.227.21:4474): protocol: context deadline exceeded; context deadline exceeded Nov 10 01:30:19.907515860 (253h49m22.609108309s) [seg_pos={0 0}, upload=223211]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 10 01:30:20.026141819 (253h49m22.727734268s) [upload=223211]: Done uploading segments... Nov 10 01:30:20.026177189 (253h49m22.727769638s) [upload=223211]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.2947266s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)