Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 3.522566613s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:04:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.522590816s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.806305732s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:03:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.806396558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................X..................X......X..X....] duration: 3.079851521s trace: https://jaeger.dev.storj.io/trace/52937cf46235962 FAIL (2026-03-10 19:03:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10 19:00:39.413986535 (2h49m42.429656142s) [upload=2723]: Done starting upload Mar 10 19:00:39.414208622 (2h49m42.429878214s) [upload=2723]: Uploading segments... Mar 10 19:00:39.414443755 (2h49m42.430113341s) [upload=2723]: Got next segment. Inline: false Mar 10 19:00:39.414502340 (2h49m42.430171927s) [upload=2723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:00:39.414508768 (2h49m42.430178375s) [upload=2723]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:00:39.527867113 (2h49m42.543536713s) [seg_pos={0 0}, upload=2723]: Begin upload segment... Mar 10 19:00:39.527946502 (2h49m42.543616108s) [noise=true , address=45.91.116.11:4131 , piece_id=Y55UCVUPK5WB6BNXKV5RS6VT3KOEXPBVM46NWRVYIY22DURCR2AA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.530040419 (2h49m42.54571002s) [seg_pos={0 0}, upload=2723]: Done begin upload segment. Mar 10 19:00:39.530134151 (2h49m42.54580376s) [noise=true , address=79.127.219.38:4838 , piece_id=3JXWAQWXM24XIXXGR4FOQPRQ2KP3HU2LJEP23ODZVGUA6ZTRWW4Q, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.530766536 (2h49m42.546436139s) [noise=true , address=79.127.219.41:4944 , piece_id=OHTCK3M2NSIKZYZ4TX2EJWU75IP4ATE5HVSPKIXUW4Q2GOPXYWCQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.531504815 (2h49m42.547174426s) [noise=true , address=193.149.164.66:4982 , piece_id=AUHO4AZLNXDGKSQCVUSIA44ANRDIHLETX4WJMIGFMRPUT3DA4AVQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.531983560 (2h49m42.547653161s) [noise=true , address=77.90.31.11:4571 , piece_id=XY35BNX2TQSTH3IOK4IMC3YZ3AZE7KLEIFJKBDGXT2P3OFWUAIOA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.532079967 (2h49m42.547749571s) [noise=true , address=5.231.73.5:4023 , piece_id=XECVDSV7MSOONC7QO6INO26JGXDDHNVMD2YU6TCVJNXLRWOO6J5Q, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.532683162 (2h49m42.548352767s) [noise=true , address=145.79.227.31:4830 , piece_id=DKQ72XLVYD3G2A6IB4D54Y55QQQHRXTR3HN2H2KOZXLQWR5QWQIQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.533246411 (2h49m42.548916013s) [noise=true , address=193.149.164.71:4287 , piece_id=TTNMMHSPVOZ2GQIYFPTETAOO53CQ3I5GRD53U3Y4UNYF6P5G2E3A, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.533803052 (2h49m42.549472656s) [noise=true , address=45.91.116.14:4198 , piece_id=6B7TI5PFCZBRZHY65YZ2TPYXJQ3IH22U5KI2N6S22H5TCUPH7UFA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.534558471 (2h49m42.550228073s) [noise=true , address=77.90.31.14:4693 , piece_id=S5OWI2MYBH46EAMBP24KHHVCXZTVNK2OXSEY7D5JY7ENOL7ALEIQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.535150911 (2h49m42.550820513s) [noise=true , address=77.90.31.17:4790 , piece_id=RUUL5GMJU5MNIL275LOA6VNPCM4YIW55PYVZ7UBPG6KV3MBGQBYQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.535407219 (2h49m42.551076828s) [noise=true , address=193.149.164.68:4034 , piece_id=H7UR3NFTZPRQHX7XB3HFTZMQJTNKAZXIEPEOGLIQUXPNMP4X4WCQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.536112245 (2h49m42.551781851s) [noise=true , address=77.90.31.18:4831 , piece_id=XTXC2H5NI5OL3N2MHMH4ZHN6F2AK4A7JSYMRTEUU5Y3MSLVWICWA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.536835429 (2h49m42.552505032s) [noise=true , address=167.17.72.71:4789 , piece_id=KRLPV4XHZ65C7GEPOJMNTFQJSDPY4CZDEH4XSNQG6KH7K6JMYJDA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.537279413 (2h49m42.552949022s) [noise=true , address=77.90.31.13:4630 , piece_id=R3JFGZVOLM73QJDCA376VDLIVCMR2ZGIWKSAPNFN4DLW2FTETT7A, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.537417577 (2h49m42.553087177s) [noise=true , address=167.17.72.70:4772 , piece_id=LXCQBLSY2RHXVNVFQN3SWOOWWNXAZ2LVLFZIAEV56WFDCN5SPNPQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.538141993 (2h49m42.553811596s) [noise=true , address=145.79.227.12:4798 , piece_id=NAMC5H7TB5BBXCGIMTPHSYGEGC56LDBVFZHCU2BJBJII5FJCJLJQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.538953556 (2h49m42.554623166s) [noise=true , address=167.17.72.69:4732 , piece_id=CG2K5EVSPJR54SAW6GMROPSPSH6RAR6VFD6DQJLTP5BF7233P7DA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:40.088124983 (2h49m43.103794586s) [upload=2723]: Next returned nil segment Mar 10 19:00:40.088147297 (2h49m43.103816903s) [upload=2723]: Waiting for error group managing segments... Mar 10 19:00:40.237376531 (2h49m43.253046139s) [noise=true , address=45.91.116.14:4198 , piece_id=6B7TI5PFCZBRZHY65YZ2TPYXJQ3IH22U5KI2N6S22H5TCUPH7UFA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.256295496 (2h49m43.271965107s) [noise=true , address=45.91.116.11:4131 , piece_id=Y55UCVUPK5WB6BNXKV5RS6VT3KOEXPBVM46NWRVYIY22DURCR2AA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.256980267 (2h49m43.272649875s) [noise=true , address=79.127.219.41:4944 , piece_id=OHTCK3M2NSIKZYZ4TX2EJWU75IP4ATE5HVSPKIXUW4Q2GOPXYWCQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.265635647 (2h49m43.281305247s) [noise=true , address=79.127.219.38:4838 , piece_id=3JXWAQWXM24XIXXGR4FOQPRQ2KP3HU2LJEP23ODZVGUA6ZTRWW4Q, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.290122587 (2h49m43.305792187s) [noise=true , address=145.79.227.12:4798 , piece_id=NAMC5H7TB5BBXCGIMTPHSYGEGC56LDBVFZHCU2BJBJII5FJCJLJQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.296604310 (2h49m43.31227391s) [noise=true , address=145.79.227.31:4830 , piece_id=DKQ72XLVYD3G2A6IB4D54Y55QQQHRXTR3HN2H2KOZXLQWR5QWQIQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.494348920 (2h49m43.510018521s) [noise=true , address=5.231.73.5:4023 , piece_id=XECVDSV7MSOONC7QO6INO26JGXDDHNVMD2YU6TCVJNXLRWOO6J5Q, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.601407556 (2h49m43.617077156s) [noise=true , address=167.17.72.69:4732 , piece_id=CG2K5EVSPJR54SAW6GMROPSPSH6RAR6VFD6DQJLTP5BF7233P7DA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.612732233 (2h49m43.628401839s) [noise=true , address=167.17.72.71:4789 , piece_id=KRLPV4XHZ65C7GEPOJMNTFQJSDPY4CZDEH4XSNQG6KH7K6JMYJDA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.751161123 (2h49m43.766830726s) [noise=true , address=193.149.164.66:4982 , piece_id=AUHO4AZLNXDGKSQCVUSIA44ANRDIHLETX4WJMIGFMRPUT3DA4AVQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.782337213 (2h49m43.798006922s) [noise=true , address=193.149.164.71:4287 , piece_id=TTNMMHSPVOZ2GQIYFPTETAOO53CQ3I5GRD53U3Y4UNYF6P5G2E3A, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.790044627 (2h49m43.805714249s) [noise=true , address=193.149.164.68:4034 , piece_id=H7UR3NFTZPRQHX7XB3HFTZMQJTNKAZXIEPEOGLIQUXPNMP4X4WCQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.887418135 (2h49m43.903087735s) [noise=true , address=167.17.72.70:4772 , piece_id=LXCQBLSY2RHXVNVFQN3SWOOWWNXAZ2LVLFZIAEV56WFDCN5SPNPQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:01:39.128684483 (2h50m42.144354085s) [noise=true , address=77.90.31.11:4571 , piece_id=XY35BNX2TQSTH3IOK4IMC3YZ3AZE7KLEIFJKBDGXT2P3OFWUAIOA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: write tcp 139.59.144.22:42740->77.90.31.11:4571: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.128963763 (2h50m42.144633372s) [noise=true , address=77.90.31.14:4693 , piece_id=S5OWI2MYBH46EAMBP24KHHVCXZTVNK2OXSEY7D5JY7ENOL7ALEIQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey, address:77.90.31.14:4693): protocol: rpc: dial tcp 77.90.31.14:4693: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129010724 (2h50m42.144680334s) [noise=true , address=77.90.31.13:4630 , piece_id=R3JFGZVOLM73QJDCA376VDLIVCMR2ZGIWKSAPNFN4DLW2FTETT7A, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129164148 (2h50m42.144833734s) [noise=true , address=77.90.31.17:4790 , piece_id=RUUL5GMJU5MNIL275LOA6VNPCM4YIW55PYVZ7UBPG6KV3MBGQBYQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129172704 (2h50m42.14484231s) [noise=true , address=77.90.31.18:4831 , piece_id=XTXC2H5NI5OL3N2MHMH4ZHN6F2AK4A7JSYMRTEUU5Y3MSLVWICWA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: write tcp 139.59.144.22:57296->77.90.31.18:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129213142 (2h50m42.144882729s) [seg_pos={0 0}, upload=2723]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:01:39.234685606 (2h50m42.250355207s) [upload=2723]: Done uploading segments... Mar 10 19:01:39.234726529 (2h50m42.25039613s) [upload=2723]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.079890557s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.820715217s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:03:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.82073904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X..................X......X..X...] duration: 3.235483816s trace: https://jaeger.dev.storj.io/trace/52937cf46235962 FAIL (2026-03-10 19:03:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10 19:00:39.413986535 (2h49m42.429656142s) [upload=2723]: Done starting upload Mar 10 19:00:39.414208622 (2h49m42.429878214s) [upload=2723]: Uploading segments... Mar 10 19:00:39.414443755 (2h49m42.430113341s) [upload=2723]: Got next segment. Inline: false Mar 10 19:00:39.414502340 (2h49m42.430171927s) [upload=2723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:00:39.414508768 (2h49m42.430178375s) [upload=2723]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:00:39.527867113 (2h49m42.543536713s) [seg_pos={0 0}, upload=2723]: Begin upload segment... Mar 10 19:00:39.527946502 (2h49m42.543616108s) [noise=true , address=45.91.116.11:4131 , piece_id=Y55UCVUPK5WB6BNXKV5RS6VT3KOEXPBVM46NWRVYIY22DURCR2AA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.530040419 (2h49m42.54571002s) [seg_pos={0 0}, upload=2723]: Done begin upload segment. Mar 10 19:00:39.530134151 (2h49m42.54580376s) [noise=true , address=79.127.219.38:4838 , piece_id=3JXWAQWXM24XIXXGR4FOQPRQ2KP3HU2LJEP23ODZVGUA6ZTRWW4Q, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.530766536 (2h49m42.546436139s) [noise=true , address=79.127.219.41:4944 , piece_id=OHTCK3M2NSIKZYZ4TX2EJWU75IP4ATE5HVSPKIXUW4Q2GOPXYWCQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.531504815 (2h49m42.547174426s) [noise=true , address=193.149.164.66:4982 , piece_id=AUHO4AZLNXDGKSQCVUSIA44ANRDIHLETX4WJMIGFMRPUT3DA4AVQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.531983560 (2h49m42.547653161s) [noise=true , address=77.90.31.11:4571 , piece_id=XY35BNX2TQSTH3IOK4IMC3YZ3AZE7KLEIFJKBDGXT2P3OFWUAIOA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.532079967 (2h49m42.547749571s) [noise=true , address=5.231.73.5:4023 , piece_id=XECVDSV7MSOONC7QO6INO26JGXDDHNVMD2YU6TCVJNXLRWOO6J5Q, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.532683162 (2h49m42.548352767s) [noise=true , address=145.79.227.31:4830 , piece_id=DKQ72XLVYD3G2A6IB4D54Y55QQQHRXTR3HN2H2KOZXLQWR5QWQIQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.533246411 (2h49m42.548916013s) [noise=true , address=193.149.164.71:4287 , piece_id=TTNMMHSPVOZ2GQIYFPTETAOO53CQ3I5GRD53U3Y4UNYF6P5G2E3A, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.533803052 (2h49m42.549472656s) [noise=true , address=45.91.116.14:4198 , piece_id=6B7TI5PFCZBRZHY65YZ2TPYXJQ3IH22U5KI2N6S22H5TCUPH7UFA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.534558471 (2h49m42.550228073s) [noise=true , address=77.90.31.14:4693 , piece_id=S5OWI2MYBH46EAMBP24KHHVCXZTVNK2OXSEY7D5JY7ENOL7ALEIQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.535150911 (2h49m42.550820513s) [noise=true , address=77.90.31.17:4790 , piece_id=RUUL5GMJU5MNIL275LOA6VNPCM4YIW55PYVZ7UBPG6KV3MBGQBYQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.535407219 (2h49m42.551076828s) [noise=true , address=193.149.164.68:4034 , piece_id=H7UR3NFTZPRQHX7XB3HFTZMQJTNKAZXIEPEOGLIQUXPNMP4X4WCQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.536112245 (2h49m42.551781851s) [noise=true , address=77.90.31.18:4831 , piece_id=XTXC2H5NI5OL3N2MHMH4ZHN6F2AK4A7JSYMRTEUU5Y3MSLVWICWA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.536835429 (2h49m42.552505032s) [noise=true , address=167.17.72.71:4789 , piece_id=KRLPV4XHZ65C7GEPOJMNTFQJSDPY4CZDEH4XSNQG6KH7K6JMYJDA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.537279413 (2h49m42.552949022s) [noise=true , address=77.90.31.13:4630 , piece_id=R3JFGZVOLM73QJDCA376VDLIVCMR2ZGIWKSAPNFN4DLW2FTETT7A, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.537417577 (2h49m42.553087177s) [noise=true , address=167.17.72.70:4772 , piece_id=LXCQBLSY2RHXVNVFQN3SWOOWWNXAZ2LVLFZIAEV56WFDCN5SPNPQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.538141993 (2h49m42.553811596s) [noise=true , address=145.79.227.12:4798 , piece_id=NAMC5H7TB5BBXCGIMTPHSYGEGC56LDBVFZHCU2BJBJII5FJCJLJQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.538953556 (2h49m42.554623166s) [noise=true , address=167.17.72.69:4732 , piece_id=CG2K5EVSPJR54SAW6GMROPSPSH6RAR6VFD6DQJLTP5BF7233P7DA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:40.088124983 (2h49m43.103794586s) [upload=2723]: Next returned nil segment Mar 10 19:00:40.088147297 (2h49m43.103816903s) [upload=2723]: Waiting for error group managing segments... Mar 10 19:00:40.237376531 (2h49m43.253046139s) [noise=true , address=45.91.116.14:4198 , piece_id=6B7TI5PFCZBRZHY65YZ2TPYXJQ3IH22U5KI2N6S22H5TCUPH7UFA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.256295496 (2h49m43.271965107s) [noise=true , address=45.91.116.11:4131 , piece_id=Y55UCVUPK5WB6BNXKV5RS6VT3KOEXPBVM46NWRVYIY22DURCR2AA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.256980267 (2h49m43.272649875s) [noise=true , address=79.127.219.41:4944 , piece_id=OHTCK3M2NSIKZYZ4TX2EJWU75IP4ATE5HVSPKIXUW4Q2GOPXYWCQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.265635647 (2h49m43.281305247s) [noise=true , address=79.127.219.38:4838 , piece_id=3JXWAQWXM24XIXXGR4FOQPRQ2KP3HU2LJEP23ODZVGUA6ZTRWW4Q, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.290122587 (2h49m43.305792187s) [noise=true , address=145.79.227.12:4798 , piece_id=NAMC5H7TB5BBXCGIMTPHSYGEGC56LDBVFZHCU2BJBJII5FJCJLJQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.296604310 (2h49m43.31227391s) [noise=true , address=145.79.227.31:4830 , piece_id=DKQ72XLVYD3G2A6IB4D54Y55QQQHRXTR3HN2H2KOZXLQWR5QWQIQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.494348920 (2h49m43.510018521s) [noise=true , address=5.231.73.5:4023 , piece_id=XECVDSV7MSOONC7QO6INO26JGXDDHNVMD2YU6TCVJNXLRWOO6J5Q, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.601407556 (2h49m43.617077156s) [noise=true , address=167.17.72.69:4732 , piece_id=CG2K5EVSPJR54SAW6GMROPSPSH6RAR6VFD6DQJLTP5BF7233P7DA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.612732233 (2h49m43.628401839s) [noise=true , address=167.17.72.71:4789 , piece_id=KRLPV4XHZ65C7GEPOJMNTFQJSDPY4CZDEH4XSNQG6KH7K6JMYJDA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.751161123 (2h49m43.766830726s) [noise=true , address=193.149.164.66:4982 , piece_id=AUHO4AZLNXDGKSQCVUSIA44ANRDIHLETX4WJMIGFMRPUT3DA4AVQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.782337213 (2h49m43.798006922s) [noise=true , address=193.149.164.71:4287 , piece_id=TTNMMHSPVOZ2GQIYFPTETAOO53CQ3I5GRD53U3Y4UNYF6P5G2E3A, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.790044627 (2h49m43.805714249s) [noise=true , address=193.149.164.68:4034 , piece_id=H7UR3NFTZPRQHX7XB3HFTZMQJTNKAZXIEPEOGLIQUXPNMP4X4WCQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.887418135 (2h49m43.903087735s) [noise=true , address=167.17.72.70:4772 , piece_id=LXCQBLSY2RHXVNVFQN3SWOOWWNXAZ2LVLFZIAEV56WFDCN5SPNPQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:01:39.128684483 (2h50m42.144354085s) [noise=true , address=77.90.31.11:4571 , piece_id=XY35BNX2TQSTH3IOK4IMC3YZ3AZE7KLEIFJKBDGXT2P3OFWUAIOA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: write tcp 139.59.144.22:42740->77.90.31.11:4571: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.128963763 (2h50m42.144633372s) [noise=true , address=77.90.31.14:4693 , piece_id=S5OWI2MYBH46EAMBP24KHHVCXZTVNK2OXSEY7D5JY7ENOL7ALEIQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey, address:77.90.31.14:4693): protocol: rpc: dial tcp 77.90.31.14:4693: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129010724 (2h50m42.144680334s) [noise=true , address=77.90.31.13:4630 , piece_id=R3JFGZVOLM73QJDCA376VDLIVCMR2ZGIWKSAPNFN4DLW2FTETT7A, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129164148 (2h50m42.144833734s) [noise=true , address=77.90.31.17:4790 , piece_id=RUUL5GMJU5MNIL275LOA6VNPCM4YIW55PYVZ7UBPG6KV3MBGQBYQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129172704 (2h50m42.14484231s) [noise=true , address=77.90.31.18:4831 , piece_id=XTXC2H5NI5OL3N2MHMH4ZHN6F2AK4A7JSYMRTEUU5Y3MSLVWICWA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: write tcp 139.59.144.22:57296->77.90.31.18:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129213142 (2h50m42.144882729s) [seg_pos={0 0}, upload=2723]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:01:39.234685606 (2h50m42.250355207s) [upload=2723]: Done uploading segments... Mar 10 19:01:39.234726529 (2h50m42.25039613s) [upload=2723]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.23551631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 1.774880901s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 19:03:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.774917107s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.710227313s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:02:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.710255423s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................X..................X......X..X..] duration: 3.332679618s trace: https://jaeger.dev.storj.io/trace/52937cf46235962 FAIL (2026-03-10 19:02:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10 19:00:39.413986535 (2h49m42.429656142s) [upload=2723]: Done starting upload Mar 10 19:00:39.414208622 (2h49m42.429878214s) [upload=2723]: Uploading segments... Mar 10 19:00:39.414443755 (2h49m42.430113341s) [upload=2723]: Got next segment. Inline: false Mar 10 19:00:39.414502340 (2h49m42.430171927s) [upload=2723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:00:39.414508768 (2h49m42.430178375s) [upload=2723]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:00:39.527867113 (2h49m42.543536713s) [seg_pos={0 0}, upload=2723]: Begin upload segment... Mar 10 19:00:39.527946502 (2h49m42.543616108s) [noise=true , address=45.91.116.11:4131 , piece_id=Y55UCVUPK5WB6BNXKV5RS6VT3KOEXPBVM46NWRVYIY22DURCR2AA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.530040419 (2h49m42.54571002s) [seg_pos={0 0}, upload=2723]: Done begin upload segment. Mar 10 19:00:39.530134151 (2h49m42.54580376s) [noise=true , address=79.127.219.38:4838 , piece_id=3JXWAQWXM24XIXXGR4FOQPRQ2KP3HU2LJEP23ODZVGUA6ZTRWW4Q, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.530766536 (2h49m42.546436139s) [noise=true , address=79.127.219.41:4944 , piece_id=OHTCK3M2NSIKZYZ4TX2EJWU75IP4ATE5HVSPKIXUW4Q2GOPXYWCQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.531504815 (2h49m42.547174426s) [noise=true , address=193.149.164.66:4982 , piece_id=AUHO4AZLNXDGKSQCVUSIA44ANRDIHLETX4WJMIGFMRPUT3DA4AVQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.531983560 (2h49m42.547653161s) [noise=true , address=77.90.31.11:4571 , piece_id=XY35BNX2TQSTH3IOK4IMC3YZ3AZE7KLEIFJKBDGXT2P3OFWUAIOA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.532079967 (2h49m42.547749571s) [noise=true , address=5.231.73.5:4023 , piece_id=XECVDSV7MSOONC7QO6INO26JGXDDHNVMD2YU6TCVJNXLRWOO6J5Q, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.532683162 (2h49m42.548352767s) [noise=true , address=145.79.227.31:4830 , piece_id=DKQ72XLVYD3G2A6IB4D54Y55QQQHRXTR3HN2H2KOZXLQWR5QWQIQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.533246411 (2h49m42.548916013s) [noise=true , address=193.149.164.71:4287 , piece_id=TTNMMHSPVOZ2GQIYFPTETAOO53CQ3I5GRD53U3Y4UNYF6P5G2E3A, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.533803052 (2h49m42.549472656s) [noise=true , address=45.91.116.14:4198 , piece_id=6B7TI5PFCZBRZHY65YZ2TPYXJQ3IH22U5KI2N6S22H5TCUPH7UFA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.534558471 (2h49m42.550228073s) [noise=true , address=77.90.31.14:4693 , piece_id=S5OWI2MYBH46EAMBP24KHHVCXZTVNK2OXSEY7D5JY7ENOL7ALEIQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.535150911 (2h49m42.550820513s) [noise=true , address=77.90.31.17:4790 , piece_id=RUUL5GMJU5MNIL275LOA6VNPCM4YIW55PYVZ7UBPG6KV3MBGQBYQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.535407219 (2h49m42.551076828s) [noise=true , address=193.149.164.68:4034 , piece_id=H7UR3NFTZPRQHX7XB3HFTZMQJTNKAZXIEPEOGLIQUXPNMP4X4WCQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.536112245 (2h49m42.551781851s) [noise=true , address=77.90.31.18:4831 , piece_id=XTXC2H5NI5OL3N2MHMH4ZHN6F2AK4A7JSYMRTEUU5Y3MSLVWICWA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.536835429 (2h49m42.552505032s) [noise=true , address=167.17.72.71:4789 , piece_id=KRLPV4XHZ65C7GEPOJMNTFQJSDPY4CZDEH4XSNQG6KH7K6JMYJDA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.537279413 (2h49m42.552949022s) [noise=true , address=77.90.31.13:4630 , piece_id=R3JFGZVOLM73QJDCA376VDLIVCMR2ZGIWKSAPNFN4DLW2FTETT7A, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.537417577 (2h49m42.553087177s) [noise=true , address=167.17.72.70:4772 , piece_id=LXCQBLSY2RHXVNVFQN3SWOOWWNXAZ2LVLFZIAEV56WFDCN5SPNPQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.538141993 (2h49m42.553811596s) [noise=true , address=145.79.227.12:4798 , piece_id=NAMC5H7TB5BBXCGIMTPHSYGEGC56LDBVFZHCU2BJBJII5FJCJLJQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.538953556 (2h49m42.554623166s) [noise=true , address=167.17.72.69:4732 , piece_id=CG2K5EVSPJR54SAW6GMROPSPSH6RAR6VFD6DQJLTP5BF7233P7DA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:40.088124983 (2h49m43.103794586s) [upload=2723]: Next returned nil segment Mar 10 19:00:40.088147297 (2h49m43.103816903s) [upload=2723]: Waiting for error group managing segments... Mar 10 19:00:40.237376531 (2h49m43.253046139s) [noise=true , address=45.91.116.14:4198 , piece_id=6B7TI5PFCZBRZHY65YZ2TPYXJQ3IH22U5KI2N6S22H5TCUPH7UFA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.256295496 (2h49m43.271965107s) [noise=true , address=45.91.116.11:4131 , piece_id=Y55UCVUPK5WB6BNXKV5RS6VT3KOEXPBVM46NWRVYIY22DURCR2AA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.256980267 (2h49m43.272649875s) [noise=true , address=79.127.219.41:4944 , piece_id=OHTCK3M2NSIKZYZ4TX2EJWU75IP4ATE5HVSPKIXUW4Q2GOPXYWCQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.265635647 (2h49m43.281305247s) [noise=true , address=79.127.219.38:4838 , piece_id=3JXWAQWXM24XIXXGR4FOQPRQ2KP3HU2LJEP23ODZVGUA6ZTRWW4Q, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.290122587 (2h49m43.305792187s) [noise=true , address=145.79.227.12:4798 , piece_id=NAMC5H7TB5BBXCGIMTPHSYGEGC56LDBVFZHCU2BJBJII5FJCJLJQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.296604310 (2h49m43.31227391s) [noise=true , address=145.79.227.31:4830 , piece_id=DKQ72XLVYD3G2A6IB4D54Y55QQQHRXTR3HN2H2KOZXLQWR5QWQIQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.494348920 (2h49m43.510018521s) [noise=true , address=5.231.73.5:4023 , piece_id=XECVDSV7MSOONC7QO6INO26JGXDDHNVMD2YU6TCVJNXLRWOO6J5Q, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.601407556 (2h49m43.617077156s) [noise=true , address=167.17.72.69:4732 , piece_id=CG2K5EVSPJR54SAW6GMROPSPSH6RAR6VFD6DQJLTP5BF7233P7DA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.612732233 (2h49m43.628401839s) [noise=true , address=167.17.72.71:4789 , piece_id=KRLPV4XHZ65C7GEPOJMNTFQJSDPY4CZDEH4XSNQG6KH7K6JMYJDA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.751161123 (2h49m43.766830726s) [noise=true , address=193.149.164.66:4982 , piece_id=AUHO4AZLNXDGKSQCVUSIA44ANRDIHLETX4WJMIGFMRPUT3DA4AVQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.782337213 (2h49m43.798006922s) [noise=true , address=193.149.164.71:4287 , piece_id=TTNMMHSPVOZ2GQIYFPTETAOO53CQ3I5GRD53U3Y4UNYF6P5G2E3A, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.790044627 (2h49m43.805714249s) [noise=true , address=193.149.164.68:4034 , piece_id=H7UR3NFTZPRQHX7XB3HFTZMQJTNKAZXIEPEOGLIQUXPNMP4X4WCQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.887418135 (2h49m43.903087735s) [noise=true , address=167.17.72.70:4772 , piece_id=LXCQBLSY2RHXVNVFQN3SWOOWWNXAZ2LVLFZIAEV56WFDCN5SPNPQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:01:39.128684483 (2h50m42.144354085s) [noise=true , address=77.90.31.11:4571 , piece_id=XY35BNX2TQSTH3IOK4IMC3YZ3AZE7KLEIFJKBDGXT2P3OFWUAIOA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: write tcp 139.59.144.22:42740->77.90.31.11:4571: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.128963763 (2h50m42.144633372s) [noise=true , address=77.90.31.14:4693 , piece_id=S5OWI2MYBH46EAMBP24KHHVCXZTVNK2OXSEY7D5JY7ENOL7ALEIQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey, address:77.90.31.14:4693): protocol: rpc: dial tcp 77.90.31.14:4693: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129010724 (2h50m42.144680334s) [noise=true , address=77.90.31.13:4630 , piece_id=R3JFGZVOLM73QJDCA376VDLIVCMR2ZGIWKSAPNFN4DLW2FTETT7A, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129164148 (2h50m42.144833734s) [noise=true , address=77.90.31.17:4790 , piece_id=RUUL5GMJU5MNIL275LOA6VNPCM4YIW55PYVZ7UBPG6KV3MBGQBYQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129172704 (2h50m42.14484231s) [noise=true , address=77.90.31.18:4831 , piece_id=XTXC2H5NI5OL3N2MHMH4ZHN6F2AK4A7JSYMRTEUU5Y3MSLVWICWA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: write tcp 139.59.144.22:57296->77.90.31.18:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129213142 (2h50m42.144882729s) [seg_pos={0 0}, upload=2723]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:01:39.234685606 (2h50m42.250355207s) [upload=2723]: Done uploading segments... Mar 10 19:01:39.234726529 (2h50m42.25039613s) [upload=2723]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.332709128s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.786596135s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:02:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.786621112s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 2.050249751s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 19:02:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.050273591s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.856177502s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:01:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.856228206s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 2.12187901s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 19:01:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.121900019s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X..................X......X..X.] duration: 3.168586788s trace: https://jaeger.dev.storj.io/trace/52937cf46235962 FAIL (2026-03-10 19:01:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10 19:00:39.413986535 (2h49m42.429656142s) [upload=2723]: Done starting upload Mar 10 19:00:39.414208622 (2h49m42.429878214s) [upload=2723]: Uploading segments... Mar 10 19:00:39.414443755 (2h49m42.430113341s) [upload=2723]: Got next segment. Inline: false Mar 10 19:00:39.414502340 (2h49m42.430171927s) [upload=2723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:00:39.414508768 (2h49m42.430178375s) [upload=2723]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:00:39.527867113 (2h49m42.543536713s) [seg_pos={0 0}, upload=2723]: Begin upload segment... Mar 10 19:00:39.527946502 (2h49m42.543616108s) [noise=true , address=45.91.116.11:4131 , piece_id=Y55UCVUPK5WB6BNXKV5RS6VT3KOEXPBVM46NWRVYIY22DURCR2AA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.530040419 (2h49m42.54571002s) [seg_pos={0 0}, upload=2723]: Done begin upload segment. Mar 10 19:00:39.530134151 (2h49m42.54580376s) [noise=true , address=79.127.219.38:4838 , piece_id=3JXWAQWXM24XIXXGR4FOQPRQ2KP3HU2LJEP23ODZVGUA6ZTRWW4Q, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.530766536 (2h49m42.546436139s) [noise=true , address=79.127.219.41:4944 , piece_id=OHTCK3M2NSIKZYZ4TX2EJWU75IP4ATE5HVSPKIXUW4Q2GOPXYWCQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.531504815 (2h49m42.547174426s) [noise=true , address=193.149.164.66:4982 , piece_id=AUHO4AZLNXDGKSQCVUSIA44ANRDIHLETX4WJMIGFMRPUT3DA4AVQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.531983560 (2h49m42.547653161s) [noise=true , address=77.90.31.11:4571 , piece_id=XY35BNX2TQSTH3IOK4IMC3YZ3AZE7KLEIFJKBDGXT2P3OFWUAIOA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.532079967 (2h49m42.547749571s) [noise=true , address=5.231.73.5:4023 , piece_id=XECVDSV7MSOONC7QO6INO26JGXDDHNVMD2YU6TCVJNXLRWOO6J5Q, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.532683162 (2h49m42.548352767s) [noise=true , address=145.79.227.31:4830 , piece_id=DKQ72XLVYD3G2A6IB4D54Y55QQQHRXTR3HN2H2KOZXLQWR5QWQIQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.533246411 (2h49m42.548916013s) [noise=true , address=193.149.164.71:4287 , piece_id=TTNMMHSPVOZ2GQIYFPTETAOO53CQ3I5GRD53U3Y4UNYF6P5G2E3A, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.533803052 (2h49m42.549472656s) [noise=true , address=45.91.116.14:4198 , piece_id=6B7TI5PFCZBRZHY65YZ2TPYXJQ3IH22U5KI2N6S22H5TCUPH7UFA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.534558471 (2h49m42.550228073s) [noise=true , address=77.90.31.14:4693 , piece_id=S5OWI2MYBH46EAMBP24KHHVCXZTVNK2OXSEY7D5JY7ENOL7ALEIQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.535150911 (2h49m42.550820513s) [noise=true , address=77.90.31.17:4790 , piece_id=RUUL5GMJU5MNIL275LOA6VNPCM4YIW55PYVZ7UBPG6KV3MBGQBYQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.535407219 (2h49m42.551076828s) [noise=true , address=193.149.164.68:4034 , piece_id=H7UR3NFTZPRQHX7XB3HFTZMQJTNKAZXIEPEOGLIQUXPNMP4X4WCQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.536112245 (2h49m42.551781851s) [noise=true , address=77.90.31.18:4831 , piece_id=XTXC2H5NI5OL3N2MHMH4ZHN6F2AK4A7JSYMRTEUU5Y3MSLVWICWA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.536835429 (2h49m42.552505032s) [noise=true , address=167.17.72.71:4789 , piece_id=KRLPV4XHZ65C7GEPOJMNTFQJSDPY4CZDEH4XSNQG6KH7K6JMYJDA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.537279413 (2h49m42.552949022s) [noise=true , address=77.90.31.13:4630 , piece_id=R3JFGZVOLM73QJDCA376VDLIVCMR2ZGIWKSAPNFN4DLW2FTETT7A, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.537417577 (2h49m42.553087177s) [noise=true , address=167.17.72.70:4772 , piece_id=LXCQBLSY2RHXVNVFQN3SWOOWWNXAZ2LVLFZIAEV56WFDCN5SPNPQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.538141993 (2h49m42.553811596s) [noise=true , address=145.79.227.12:4798 , piece_id=NAMC5H7TB5BBXCGIMTPHSYGEGC56LDBVFZHCU2BJBJII5FJCJLJQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.538953556 (2h49m42.554623166s) [noise=true , address=167.17.72.69:4732 , piece_id=CG2K5EVSPJR54SAW6GMROPSPSH6RAR6VFD6DQJLTP5BF7233P7DA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:40.088124983 (2h49m43.103794586s) [upload=2723]: Next returned nil segment Mar 10 19:00:40.088147297 (2h49m43.103816903s) [upload=2723]: Waiting for error group managing segments... Mar 10 19:00:40.237376531 (2h49m43.253046139s) [noise=true , address=45.91.116.14:4198 , piece_id=6B7TI5PFCZBRZHY65YZ2TPYXJQ3IH22U5KI2N6S22H5TCUPH7UFA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.256295496 (2h49m43.271965107s) [noise=true , address=45.91.116.11:4131 , piece_id=Y55UCVUPK5WB6BNXKV5RS6VT3KOEXPBVM46NWRVYIY22DURCR2AA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.256980267 (2h49m43.272649875s) [noise=true , address=79.127.219.41:4944 , piece_id=OHTCK3M2NSIKZYZ4TX2EJWU75IP4ATE5HVSPKIXUW4Q2GOPXYWCQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.265635647 (2h49m43.281305247s) [noise=true , address=79.127.219.38:4838 , piece_id=3JXWAQWXM24XIXXGR4FOQPRQ2KP3HU2LJEP23ODZVGUA6ZTRWW4Q, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.290122587 (2h49m43.305792187s) [noise=true , address=145.79.227.12:4798 , piece_id=NAMC5H7TB5BBXCGIMTPHSYGEGC56LDBVFZHCU2BJBJII5FJCJLJQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.296604310 (2h49m43.31227391s) [noise=true , address=145.79.227.31:4830 , piece_id=DKQ72XLVYD3G2A6IB4D54Y55QQQHRXTR3HN2H2KOZXLQWR5QWQIQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.494348920 (2h49m43.510018521s) [noise=true , address=5.231.73.5:4023 , piece_id=XECVDSV7MSOONC7QO6INO26JGXDDHNVMD2YU6TCVJNXLRWOO6J5Q, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.601407556 (2h49m43.617077156s) [noise=true , address=167.17.72.69:4732 , piece_id=CG2K5EVSPJR54SAW6GMROPSPSH6RAR6VFD6DQJLTP5BF7233P7DA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.612732233 (2h49m43.628401839s) [noise=true , address=167.17.72.71:4789 , piece_id=KRLPV4XHZ65C7GEPOJMNTFQJSDPY4CZDEH4XSNQG6KH7K6JMYJDA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.751161123 (2h49m43.766830726s) [noise=true , address=193.149.164.66:4982 , piece_id=AUHO4AZLNXDGKSQCVUSIA44ANRDIHLETX4WJMIGFMRPUT3DA4AVQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.782337213 (2h49m43.798006922s) [noise=true , address=193.149.164.71:4287 , piece_id=TTNMMHSPVOZ2GQIYFPTETAOO53CQ3I5GRD53U3Y4UNYF6P5G2E3A, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.790044627 (2h49m43.805714249s) [noise=true , address=193.149.164.68:4034 , piece_id=H7UR3NFTZPRQHX7XB3HFTZMQJTNKAZXIEPEOGLIQUXPNMP4X4WCQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.887418135 (2h49m43.903087735s) [noise=true , address=167.17.72.70:4772 , piece_id=LXCQBLSY2RHXVNVFQN3SWOOWWNXAZ2LVLFZIAEV56WFDCN5SPNPQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:01:39.128684483 (2h50m42.144354085s) [noise=true , address=77.90.31.11:4571 , piece_id=XY35BNX2TQSTH3IOK4IMC3YZ3AZE7KLEIFJKBDGXT2P3OFWUAIOA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: write tcp 139.59.144.22:42740->77.90.31.11:4571: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.128963763 (2h50m42.144633372s) [noise=true , address=77.90.31.14:4693 , piece_id=S5OWI2MYBH46EAMBP24KHHVCXZTVNK2OXSEY7D5JY7ENOL7ALEIQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey, address:77.90.31.14:4693): protocol: rpc: dial tcp 77.90.31.14:4693: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129010724 (2h50m42.144680334s) [noise=true , address=77.90.31.13:4630 , piece_id=R3JFGZVOLM73QJDCA376VDLIVCMR2ZGIWKSAPNFN4DLW2FTETT7A, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129164148 (2h50m42.144833734s) [noise=true , address=77.90.31.17:4790 , piece_id=RUUL5GMJU5MNIL275LOA6VNPCM4YIW55PYVZ7UBPG6KV3MBGQBYQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129172704 (2h50m42.14484231s) [noise=true , address=77.90.31.18:4831 , piece_id=XTXC2H5NI5OL3N2MHMH4ZHN6F2AK4A7JSYMRTEUU5Y3MSLVWICWA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: write tcp 139.59.144.22:57296->77.90.31.18:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129213142 (2h50m42.144882729s) [seg_pos={0 0}, upload=2723]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:01:39.234685606 (2h50m42.250355207s) [upload=2723]: Done uploading segments... Mar 10 19:01:39.234726529 (2h50m42.25039613s) [upload=2723]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.168604817s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................X..................X......X..X] duration: 1m0.107991238s trace: https://jaeger.dev.storj.io/trace/52937cf46235962 FAIL (2026-03-10 19:01:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 10 19:00:39.413986535 (2h49m42.429656142s) [upload=2723]: Done starting upload Mar 10 19:00:39.414208622 (2h49m42.429878214s) [upload=2723]: Uploading segments... Mar 10 19:00:39.414443755 (2h49m42.430113341s) [upload=2723]: Got next segment. Inline: false Mar 10 19:00:39.414502340 (2h49m42.430171927s) [upload=2723]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 19:00:39.414508768 (2h49m42.430178375s) [upload=2723]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 19:00:39.527867113 (2h49m42.543536713s) [seg_pos={0 0}, upload=2723]: Begin upload segment... Mar 10 19:00:39.527946502 (2h49m42.543616108s) [noise=true , address=45.91.116.11:4131 , piece_id=Y55UCVUPK5WB6BNXKV5RS6VT3KOEXPBVM46NWRVYIY22DURCR2AA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.530040419 (2h49m42.54571002s) [seg_pos={0 0}, upload=2723]: Done begin upload segment. Mar 10 19:00:39.530134151 (2h49m42.54580376s) [noise=true , address=79.127.219.38:4838 , piece_id=3JXWAQWXM24XIXXGR4FOQPRQ2KP3HU2LJEP23ODZVGUA6ZTRWW4Q, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.530766536 (2h49m42.546436139s) [noise=true , address=79.127.219.41:4944 , piece_id=OHTCK3M2NSIKZYZ4TX2EJWU75IP4ATE5HVSPKIXUW4Q2GOPXYWCQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.531504815 (2h49m42.547174426s) [noise=true , address=193.149.164.66:4982 , piece_id=AUHO4AZLNXDGKSQCVUSIA44ANRDIHLETX4WJMIGFMRPUT3DA4AVQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.531983560 (2h49m42.547653161s) [noise=true , address=77.90.31.11:4571 , piece_id=XY35BNX2TQSTH3IOK4IMC3YZ3AZE7KLEIFJKBDGXT2P3OFWUAIOA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.532079967 (2h49m42.547749571s) [noise=true , address=5.231.73.5:4023 , piece_id=XECVDSV7MSOONC7QO6INO26JGXDDHNVMD2YU6TCVJNXLRWOO6J5Q, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.532683162 (2h49m42.548352767s) [noise=true , address=145.79.227.31:4830 , piece_id=DKQ72XLVYD3G2A6IB4D54Y55QQQHRXTR3HN2H2KOZXLQWR5QWQIQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.533246411 (2h49m42.548916013s) [noise=true , address=193.149.164.71:4287 , piece_id=TTNMMHSPVOZ2GQIYFPTETAOO53CQ3I5GRD53U3Y4UNYF6P5G2E3A, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.533803052 (2h49m42.549472656s) [noise=true , address=45.91.116.14:4198 , piece_id=6B7TI5PFCZBRZHY65YZ2TPYXJQ3IH22U5KI2N6S22H5TCUPH7UFA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.534558471 (2h49m42.550228073s) [noise=true , address=77.90.31.14:4693 , piece_id=S5OWI2MYBH46EAMBP24KHHVCXZTVNK2OXSEY7D5JY7ENOL7ALEIQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.535150911 (2h49m42.550820513s) [noise=true , address=77.90.31.17:4790 , piece_id=RUUL5GMJU5MNIL275LOA6VNPCM4YIW55PYVZ7UBPG6KV3MBGQBYQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.535407219 (2h49m42.551076828s) [noise=true , address=193.149.164.68:4034 , piece_id=H7UR3NFTZPRQHX7XB3HFTZMQJTNKAZXIEPEOGLIQUXPNMP4X4WCQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.536112245 (2h49m42.551781851s) [noise=true , address=77.90.31.18:4831 , piece_id=XTXC2H5NI5OL3N2MHMH4ZHN6F2AK4A7JSYMRTEUU5Y3MSLVWICWA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.536835429 (2h49m42.552505032s) [noise=true , address=167.17.72.71:4789 , piece_id=KRLPV4XHZ65C7GEPOJMNTFQJSDPY4CZDEH4XSNQG6KH7K6JMYJDA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.537279413 (2h49m42.552949022s) [noise=true , address=77.90.31.13:4630 , piece_id=R3JFGZVOLM73QJDCA376VDLIVCMR2ZGIWKSAPNFN4DLW2FTETT7A, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.537417577 (2h49m42.553087177s) [noise=true , address=167.17.72.70:4772 , piece_id=LXCQBLSY2RHXVNVFQN3SWOOWWNXAZ2LVLFZIAEV56WFDCN5SPNPQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.538141993 (2h49m42.553811596s) [noise=true , address=145.79.227.12:4798 , piece_id=NAMC5H7TB5BBXCGIMTPHSYGEGC56LDBVFZHCU2BJBJII5FJCJLJQ, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:39.538953556 (2h49m42.554623166s) [noise=true , address=167.17.72.69:4732 , piece_id=CG2K5EVSPJR54SAW6GMROPSPSH6RAR6VFD6DQJLTP5BF7233P7DA, seg_pos={0 0}, upload=2723]: Uploading piece... Mar 10 19:00:40.088124983 (2h49m43.103794586s) [upload=2723]: Next returned nil segment Mar 10 19:00:40.088147297 (2h49m43.103816903s) [upload=2723]: Waiting for error group managing segments... Mar 10 19:00:40.237376531 (2h49m43.253046139s) [noise=true , address=45.91.116.14:4198 , piece_id=6B7TI5PFCZBRZHY65YZ2TPYXJQ3IH22U5KI2N6S22H5TCUPH7UFA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.256295496 (2h49m43.271965107s) [noise=true , address=45.91.116.11:4131 , piece_id=Y55UCVUPK5WB6BNXKV5RS6VT3KOEXPBVM46NWRVYIY22DURCR2AA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.256980267 (2h49m43.272649875s) [noise=true , address=79.127.219.41:4944 , piece_id=OHTCK3M2NSIKZYZ4TX2EJWU75IP4ATE5HVSPKIXUW4Q2GOPXYWCQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.265635647 (2h49m43.281305247s) [noise=true , address=79.127.219.38:4838 , piece_id=3JXWAQWXM24XIXXGR4FOQPRQ2KP3HU2LJEP23ODZVGUA6ZTRWW4Q, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.290122587 (2h49m43.305792187s) [noise=true , address=145.79.227.12:4798 , piece_id=NAMC5H7TB5BBXCGIMTPHSYGEGC56LDBVFZHCU2BJBJII5FJCJLJQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.296604310 (2h49m43.31227391s) [noise=true , address=145.79.227.31:4830 , piece_id=DKQ72XLVYD3G2A6IB4D54Y55QQQHRXTR3HN2H2KOZXLQWR5QWQIQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.494348920 (2h49m43.510018521s) [noise=true , address=5.231.73.5:4023 , piece_id=XECVDSV7MSOONC7QO6INO26JGXDDHNVMD2YU6TCVJNXLRWOO6J5Q, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.601407556 (2h49m43.617077156s) [noise=true , address=167.17.72.69:4732 , piece_id=CG2K5EVSPJR54SAW6GMROPSPSH6RAR6VFD6DQJLTP5BF7233P7DA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.612732233 (2h49m43.628401839s) [noise=true , address=167.17.72.71:4789 , piece_id=KRLPV4XHZ65C7GEPOJMNTFQJSDPY4CZDEH4XSNQG6KH7K6JMYJDA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.751161123 (2h49m43.766830726s) [noise=true , address=193.149.164.66:4982 , piece_id=AUHO4AZLNXDGKSQCVUSIA44ANRDIHLETX4WJMIGFMRPUT3DA4AVQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.782337213 (2h49m43.798006922s) [noise=true , address=193.149.164.71:4287 , piece_id=TTNMMHSPVOZ2GQIYFPTETAOO53CQ3I5GRD53U3Y4UNYF6P5G2E3A, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.790044627 (2h49m43.805714249s) [noise=true , address=193.149.164.68:4034 , piece_id=H7UR3NFTZPRQHX7XB3HFTZMQJTNKAZXIEPEOGLIQUXPNMP4X4WCQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:00:40.887418135 (2h49m43.903087735s) [noise=true , address=167.17.72.70:4772 , piece_id=LXCQBLSY2RHXVNVFQN3SWOOWWNXAZ2LVLFZIAEV56WFDCN5SPNPQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: <nil> Mar 10 19:01:39.128684483 (2h50m42.144354085s) [noise=true , address=77.90.31.11:4571 , piece_id=XY35BNX2TQSTH3IOK4IMC3YZ3AZE7KLEIFJKBDGXT2P3OFWUAIOA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:12eshfvC8guQXYqSeUqwmycyBbM4ens9W98cktmW7nPzk1D6pwy, address:77.90.31.11:4571): protocol: write tcp 139.59.144.22:42740->77.90.31.11:4571: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.128963763 (2h50m42.144633372s) [noise=true , address=77.90.31.14:4693 , piece_id=S5OWI2MYBH46EAMBP24KHHVCXZTVNK2OXSEY7D5JY7ENOL7ALEIQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1RASrnvpAj4ToD4xNvbma1iKZEe2uHK8WosjfZwDDom8Qd17Ey, address:77.90.31.14:4693): protocol: rpc: dial tcp 77.90.31.14:4693: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129010724 (2h50m42.144680334s) [noise=true , address=77.90.31.13:4630 , piece_id=R3JFGZVOLM73QJDCA376VDLIVCMR2ZGIWKSAPNFN4DLW2FTETT7A, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn, address:77.90.31.13:4630): protocol: rpc: dial tcp 77.90.31.13:4630: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129164148 (2h50m42.144833734s) [noise=true , address=77.90.31.17:4790 , piece_id=RUUL5GMJU5MNIL275LOA6VNPCM4YIW55PYVZ7UBPG6KV3MBGQBYQ, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1vDiVaJCoCnC6GqF5ZjhVwA6QCo1HkvRDYqLhEfKM8z91bv25b, address:77.90.31.17:4790): protocol: rpc: dial tcp 77.90.31.17:4790: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129172704 (2h50m42.14484231s) [noise=true , address=77.90.31.18:4831 , piece_id=XTXC2H5NI5OL3N2MHMH4ZHN6F2AK4A7JSYMRTEUU5Y3MSLVWICWA, seg_pos={0 0}, upload=2723]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: write tcp 139.59.144.22:57296->77.90.31.18:4831: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 19:01:39.129213142 (2h50m42.144882729s) [seg_pos={0 0}, upload=2723]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 19:01:39.234685606 (2h50m42.250355207s) [upload=2723]: Done uploading segments... Mar 10 19:01:39.234726529 (2h50m42.25039613s) [upload=2723]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.108015737s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.78335931s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 19:00:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.783385072s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 2.248796905s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 19:00:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.248828662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X..................X......X..] duration: 3.273111364s trace: https://jaeger.dev.storj.io/trace/542a9c4646f77fb5 FAIL (2026-03-10 19:00:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 18:58:48.813308868 (2h47m51.828978477s) [upload=2692]: Done starting upload Mar 10 18:58:48.813424821 (2h47m51.829094415s) [upload=2692]: Uploading segments... Mar 10 18:58:48.813771258 (2h47m51.829440858s) [upload=2692]: Got next segment. Inline: false Mar 10 18:58:48.813822395 (2h47m51.829491999s) [upload=2692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:58:48.813828121 (2h47m51.829497722s) [upload=2692]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:58:48.944926562 (2h47m51.960596155s) [seg_pos={0 0}, upload=2692]: Begin upload segment... Mar 10 18:58:48.944983382 (2h47m51.960652985s) [seg_pos={0 0}, upload=2692]: Done begin upload segment. Mar 10 18:58:48.945013521 (2h47m51.960683126s) [noise=true , address=77.90.31.17:4778 , piece_id=EGXEOIXDD7UCWD5WP4GEZMIAVZYHDBZCKCRNQPWE6DLCMM33HXYA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.946083132 (2h47m51.961752742s) [noise=true , address=64.49.13.27:4957 , piece_id=2TZTPQ77F3WNPQABUTFYCOBWGNTBWY6WZO4BJSPPYGPXKNAU2Y6A, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.946361136 (2h47m51.962030726s) [noise=true , address=167.17.72.70:4773 , piece_id=VC6Z43LAQHKPRZHJOQBJDB3FGQUVWKLMNWYBDRKZ5MAYN74QI7YQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.946884834 (2h47m51.962554445s) [noise=true , address=79.127.219.46:4904 , piece_id=ICXF2DTBO4MHPIIOFCGRUX7XYXBGHKA6Y5LEOMVRTZPR5NMTKQLQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.947061928 (2h47m51.962731538s) [noise=true , address=77.90.31.21:4908 , piece_id=MGLX6W62QSSMBHIAXPWCUQ4GDSTB5GCHVC7V7DIORVD57FW6VCXQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.947372660 (2h47m51.963042268s) [noise=true , address=45.91.116.9:4274 , piece_id=5XTDYTSS4YMF5LTZXXGOTE74IP4O7QBWR3WXYPEJKCQ2N6H5P3AA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.947594049 (2h47m51.963263658s) [noise=true , address=79.127.205.235:4272 , piece_id=HKNWRACPWIEHUME6E2MNHYTZJ4UAST46SYMBYIE336BO5HZU35VA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.947864759 (2h47m51.96353437s) [noise=true , address=79.127.205.233:4144 , piece_id=2MAH5ZDWDUULY7CI2OX3WJJ4U2MQTW5U3YYW6J4HV6KC5WIQC4EQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.948080941 (2h47m51.963750552s) [noise=true , address=64.49.13.139:4478 , piece_id=BLIB3MQ2KR3I2THBDIP5CIATPDY656PPAEETACFG65T25YTOOOYA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.948313384 (2h47m51.963982991s) [noise=true , address=77.90.31.18:4829 , piece_id=MV6Y3DRAKTJM3C4WJ4XBCLECORU3JH47DWF7MUFSWFWE4EZ6JOHQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.948767373 (2h47m51.964436981s) [noise=true , address=77.90.31.15:4717 , piece_id=KS5ZU2QOWKQENYRO6GBTNG7MEDAADMCZQQSTP7VWER3F3ETFN6XQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.948911100 (2h47m51.964580712s) [noise=true , address=167.17.72.69:4716 , piece_id=53SRRSQC6RBKWWHB7LQDBIMUMO4CWX4XYOBVHNC6KVJNPA4ZWUZA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.949333055 (2h47m51.965002716s) [noise=true , address=45.91.116.7:4036 , piece_id=XDKWF3ITQFPJULUQQ5NLKMR7BIB3YFFB33U7NISIBYUMSB2E2FUA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.949421824 (2h47m51.965091438s) [noise=true , address=77.90.31.10:4541 , piece_id=A6FZGPQOZ4XGQ5PJANJ5NHXNBMZVRXZ76FW7MT77ZR55IB6HHURA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.949877571 (2h47m51.965547227s) [noise=true , address=5.231.73.3:4034 , piece_id=4FQEKWBSA3LOBC5Z7SDPAD6KYVGFXA2FLASNNWI7SVWMRP2CVLMQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.949954655 (2h47m51.965624263s) [noise=true , address=45.91.116.6:4002 , piece_id=7TDZ66PYN47BYXEVHI5J47OCNXVG5V44DEK4M6EEUECBUYV2RW2A, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.950451735 (2h47m51.966121344s) [noise=true , address=193.149.164.68:4068 , piece_id=5AXNZSIXZVP7QRRCNYH6WJSVFZMX4INBU4PIEMFUVD6SY6QFWDUA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.950939313 (2h47m51.966608919s) [noise=true , address=167.17.72.71:4816 , piece_id=3XQB2F5SSCIPO4RV3SF2QTGEXNZWBTD3C6IRO6AMDKBHZYROSO4A, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:49.515288752 (2h47m52.530958353s) [upload=2692]: Next returned nil segment Mar 10 18:58:49.515316263 (2h47m52.530985864s) [upload=2692]: Waiting for error group managing segments... Mar 10 18:58:49.668312744 (2h47m52.683982347s) [noise=true , address=45.91.116.7:4036 , piece_id=XDKWF3ITQFPJULUQQ5NLKMR7BIB3YFFB33U7NISIBYUMSB2E2FUA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:49.681137054 (2h47m52.696806654s) [noise=true , address=45.91.116.9:4274 , piece_id=5XTDYTSS4YMF5LTZXXGOTE74IP4O7QBWR3WXYPEJKCQ2N6H5P3AA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:49.689138627 (2h47m52.704808251s) [noise=true , address=45.91.116.6:4002 , piece_id=7TDZ66PYN47BYXEVHI5J47OCNXVG5V44DEK4M6EEUECBUYV2RW2A, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:49.717308168 (2h47m52.732977769s) [noise=true , address=79.127.219.46:4904 , piece_id=ICXF2DTBO4MHPIIOFCGRUX7XYXBGHKA6Y5LEOMVRTZPR5NMTKQLQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:49.934665513 (2h47m52.950335113s) [noise=true , address=5.231.73.3:4034 , piece_id=4FQEKWBSA3LOBC5Z7SDPAD6KYVGFXA2FLASNNWI7SVWMRP2CVLMQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.024032552 (2h47m53.039702152s) [noise=true , address=167.17.72.71:4816 , piece_id=3XQB2F5SSCIPO4RV3SF2QTGEXNZWBTD3C6IRO6AMDKBHZYROSO4A, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.041285961 (2h47m53.056955562s) [noise=true , address=167.17.72.70:4773 , piece_id=VC6Z43LAQHKPRZHJOQBJDB3FGQUVWKLMNWYBDRKZ5MAYN74QI7YQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.082585315 (2h47m53.098254916s) [noise=true , address=167.17.72.69:4716 , piece_id=53SRRSQC6RBKWWHB7LQDBIMUMO4CWX4XYOBVHNC6KVJNPA4ZWUZA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.168883082 (2h47m53.184552685s) [noise=true , address=64.49.13.27:4957 , piece_id=2TZTPQ77F3WNPQABUTFYCOBWGNTBWY6WZO4BJSPPYGPXKNAU2Y6A, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.176996442 (2h47m53.192666051s) [noise=true , address=193.149.164.68:4068 , piece_id=5AXNZSIXZVP7QRRCNYH6WJSVFZMX4INBU4PIEMFUVD6SY6QFWDUA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.205136065 (2h47m53.220805669s) [noise=true , address=64.49.13.139:4478 , piece_id=BLIB3MQ2KR3I2THBDIP5CIATPDY656PPAEETACFG65T25YTOOOYA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.236254587 (2h47m53.251924194s) [noise=true , address=79.127.205.235:4272 , piece_id=HKNWRACPWIEHUME6E2MNHYTZJ4UAST46SYMBYIE336BO5HZU35VA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.362717963 (2h47m53.37838757s) [noise=true , address=79.127.205.233:4144 , piece_id=2MAH5ZDWDUULY7CI2OX3WJJ4U2MQTW5U3YYW6J4HV6KC5WIQC4EQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:59:48.529863846 (2h48m51.54553345s) [noise=true , address=77.90.31.21:4908 , piece_id=MGLX6W62QSSMBHIAXPWCUQ4GDSTB5GCHVC7V7DIORVD57FW6VCXQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, address:77.90.31.21:4908): protocol: rpc: dial tcp 77.90.31.21:4908: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530274966 (2h48m51.545944572s) [noise=true , address=77.90.31.18:4829 , piece_id=MV6Y3DRAKTJM3C4WJ4XBCLECORU3JH47DWF7MUFSWFWE4EZ6JOHQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, address:77.90.31.18:4829): protocol: rpc: dial tcp 77.90.31.18:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530469932 (2h48m51.546139538s) [noise=true , address=77.90.31.10:4541 , piece_id=A6FZGPQOZ4XGQ5PJANJ5NHXNBMZVRXZ76FW7MT77ZR55IB6HHURA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW, address:77.90.31.10:4541): protocol: write tcp 139.59.144.22:47358->77.90.31.10:4541: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530721033 (2h48m51.546390642s) [noise=true , address=77.90.31.17:4778 , piece_id=EGXEOIXDD7UCWD5WP4GEZMIAVZYHDBZCKCRNQPWE6DLCMM33HXYA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530765398 (2h48m51.546434998s) [noise=true , address=77.90.31.15:4717 , piece_id=KS5ZU2QOWKQENYRO6GBTNG7MEDAADMCZQQSTP7VWER3F3ETFN6XQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 139.59.144.22:60646->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530789282 (2h48m51.546458882s) [seg_pos={0 0}, upload=2692]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:59:48.650623196 (2h48m51.666292797s) [upload=2692]: Done uploading segments... Mar 10 18:59:48.650672655 (2h48m51.666342258s) [upload=2692]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.273207135s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 3.400012324s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:59:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.4001974s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................X..................X......X.] duration: 3.093483151s trace: https://jaeger.dev.storj.io/trace/542a9c4646f77fb5 FAIL (2026-03-10 18:59:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 18:58:48.813308868 (2h47m51.828978477s) [upload=2692]: Done starting upload Mar 10 18:58:48.813424821 (2h47m51.829094415s) [upload=2692]: Uploading segments... Mar 10 18:58:48.813771258 (2h47m51.829440858s) [upload=2692]: Got next segment. Inline: false Mar 10 18:58:48.813822395 (2h47m51.829491999s) [upload=2692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:58:48.813828121 (2h47m51.829497722s) [upload=2692]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:58:48.944926562 (2h47m51.960596155s) [seg_pos={0 0}, upload=2692]: Begin upload segment... Mar 10 18:58:48.944983382 (2h47m51.960652985s) [seg_pos={0 0}, upload=2692]: Done begin upload segment. Mar 10 18:58:48.945013521 (2h47m51.960683126s) [noise=true , address=77.90.31.17:4778 , piece_id=EGXEOIXDD7UCWD5WP4GEZMIAVZYHDBZCKCRNQPWE6DLCMM33HXYA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.946083132 (2h47m51.961752742s) [noise=true , address=64.49.13.27:4957 , piece_id=2TZTPQ77F3WNPQABUTFYCOBWGNTBWY6WZO4BJSPPYGPXKNAU2Y6A, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.946361136 (2h47m51.962030726s) [noise=true , address=167.17.72.70:4773 , piece_id=VC6Z43LAQHKPRZHJOQBJDB3FGQUVWKLMNWYBDRKZ5MAYN74QI7YQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.946884834 (2h47m51.962554445s) [noise=true , address=79.127.219.46:4904 , piece_id=ICXF2DTBO4MHPIIOFCGRUX7XYXBGHKA6Y5LEOMVRTZPR5NMTKQLQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.947061928 (2h47m51.962731538s) [noise=true , address=77.90.31.21:4908 , piece_id=MGLX6W62QSSMBHIAXPWCUQ4GDSTB5GCHVC7V7DIORVD57FW6VCXQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.947372660 (2h47m51.963042268s) [noise=true , address=45.91.116.9:4274 , piece_id=5XTDYTSS4YMF5LTZXXGOTE74IP4O7QBWR3WXYPEJKCQ2N6H5P3AA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.947594049 (2h47m51.963263658s) [noise=true , address=79.127.205.235:4272 , piece_id=HKNWRACPWIEHUME6E2MNHYTZJ4UAST46SYMBYIE336BO5HZU35VA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.947864759 (2h47m51.96353437s) [noise=true , address=79.127.205.233:4144 , piece_id=2MAH5ZDWDUULY7CI2OX3WJJ4U2MQTW5U3YYW6J4HV6KC5WIQC4EQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.948080941 (2h47m51.963750552s) [noise=true , address=64.49.13.139:4478 , piece_id=BLIB3MQ2KR3I2THBDIP5CIATPDY656PPAEETACFG65T25YTOOOYA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.948313384 (2h47m51.963982991s) [noise=true , address=77.90.31.18:4829 , piece_id=MV6Y3DRAKTJM3C4WJ4XBCLECORU3JH47DWF7MUFSWFWE4EZ6JOHQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.948767373 (2h47m51.964436981s) [noise=true , address=77.90.31.15:4717 , piece_id=KS5ZU2QOWKQENYRO6GBTNG7MEDAADMCZQQSTP7VWER3F3ETFN6XQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.948911100 (2h47m51.964580712s) [noise=true , address=167.17.72.69:4716 , piece_id=53SRRSQC6RBKWWHB7LQDBIMUMO4CWX4XYOBVHNC6KVJNPA4ZWUZA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.949333055 (2h47m51.965002716s) [noise=true , address=45.91.116.7:4036 , piece_id=XDKWF3ITQFPJULUQQ5NLKMR7BIB3YFFB33U7NISIBYUMSB2E2FUA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.949421824 (2h47m51.965091438s) [noise=true , address=77.90.31.10:4541 , piece_id=A6FZGPQOZ4XGQ5PJANJ5NHXNBMZVRXZ76FW7MT77ZR55IB6HHURA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.949877571 (2h47m51.965547227s) [noise=true , address=5.231.73.3:4034 , piece_id=4FQEKWBSA3LOBC5Z7SDPAD6KYVGFXA2FLASNNWI7SVWMRP2CVLMQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.949954655 (2h47m51.965624263s) [noise=true , address=45.91.116.6:4002 , piece_id=7TDZ66PYN47BYXEVHI5J47OCNXVG5V44DEK4M6EEUECBUYV2RW2A, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.950451735 (2h47m51.966121344s) [noise=true , address=193.149.164.68:4068 , piece_id=5AXNZSIXZVP7QRRCNYH6WJSVFZMX4INBU4PIEMFUVD6SY6QFWDUA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.950939313 (2h47m51.966608919s) [noise=true , address=167.17.72.71:4816 , piece_id=3XQB2F5SSCIPO4RV3SF2QTGEXNZWBTD3C6IRO6AMDKBHZYROSO4A, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:49.515288752 (2h47m52.530958353s) [upload=2692]: Next returned nil segment Mar 10 18:58:49.515316263 (2h47m52.530985864s) [upload=2692]: Waiting for error group managing segments... Mar 10 18:58:49.668312744 (2h47m52.683982347s) [noise=true , address=45.91.116.7:4036 , piece_id=XDKWF3ITQFPJULUQQ5NLKMR7BIB3YFFB33U7NISIBYUMSB2E2FUA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:49.681137054 (2h47m52.696806654s) [noise=true , address=45.91.116.9:4274 , piece_id=5XTDYTSS4YMF5LTZXXGOTE74IP4O7QBWR3WXYPEJKCQ2N6H5P3AA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:49.689138627 (2h47m52.704808251s) [noise=true , address=45.91.116.6:4002 , piece_id=7TDZ66PYN47BYXEVHI5J47OCNXVG5V44DEK4M6EEUECBUYV2RW2A, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:49.717308168 (2h47m52.732977769s) [noise=true , address=79.127.219.46:4904 , piece_id=ICXF2DTBO4MHPIIOFCGRUX7XYXBGHKA6Y5LEOMVRTZPR5NMTKQLQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:49.934665513 (2h47m52.950335113s) [noise=true , address=5.231.73.3:4034 , piece_id=4FQEKWBSA3LOBC5Z7SDPAD6KYVGFXA2FLASNNWI7SVWMRP2CVLMQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.024032552 (2h47m53.039702152s) [noise=true , address=167.17.72.71:4816 , piece_id=3XQB2F5SSCIPO4RV3SF2QTGEXNZWBTD3C6IRO6AMDKBHZYROSO4A, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.041285961 (2h47m53.056955562s) [noise=true , address=167.17.72.70:4773 , piece_id=VC6Z43LAQHKPRZHJOQBJDB3FGQUVWKLMNWYBDRKZ5MAYN74QI7YQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.082585315 (2h47m53.098254916s) [noise=true , address=167.17.72.69:4716 , piece_id=53SRRSQC6RBKWWHB7LQDBIMUMO4CWX4XYOBVHNC6KVJNPA4ZWUZA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.168883082 (2h47m53.184552685s) [noise=true , address=64.49.13.27:4957 , piece_id=2TZTPQ77F3WNPQABUTFYCOBWGNTBWY6WZO4BJSPPYGPXKNAU2Y6A, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.176996442 (2h47m53.192666051s) [noise=true , address=193.149.164.68:4068 , piece_id=5AXNZSIXZVP7QRRCNYH6WJSVFZMX4INBU4PIEMFUVD6SY6QFWDUA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.205136065 (2h47m53.220805669s) [noise=true , address=64.49.13.139:4478 , piece_id=BLIB3MQ2KR3I2THBDIP5CIATPDY656PPAEETACFG65T25YTOOOYA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.236254587 (2h47m53.251924194s) [noise=true , address=79.127.205.235:4272 , piece_id=HKNWRACPWIEHUME6E2MNHYTZJ4UAST46SYMBYIE336BO5HZU35VA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.362717963 (2h47m53.37838757s) [noise=true , address=79.127.205.233:4144 , piece_id=2MAH5ZDWDUULY7CI2OX3WJJ4U2MQTW5U3YYW6J4HV6KC5WIQC4EQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:59:48.529863846 (2h48m51.54553345s) [noise=true , address=77.90.31.21:4908 , piece_id=MGLX6W62QSSMBHIAXPWCUQ4GDSTB5GCHVC7V7DIORVD57FW6VCXQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, address:77.90.31.21:4908): protocol: rpc: dial tcp 77.90.31.21:4908: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530274966 (2h48m51.545944572s) [noise=true , address=77.90.31.18:4829 , piece_id=MV6Y3DRAKTJM3C4WJ4XBCLECORU3JH47DWF7MUFSWFWE4EZ6JOHQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, address:77.90.31.18:4829): protocol: rpc: dial tcp 77.90.31.18:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530469932 (2h48m51.546139538s) [noise=true , address=77.90.31.10:4541 , piece_id=A6FZGPQOZ4XGQ5PJANJ5NHXNBMZVRXZ76FW7MT77ZR55IB6HHURA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW, address:77.90.31.10:4541): protocol: write tcp 139.59.144.22:47358->77.90.31.10:4541: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530721033 (2h48m51.546390642s) [noise=true , address=77.90.31.17:4778 , piece_id=EGXEOIXDD7UCWD5WP4GEZMIAVZYHDBZCKCRNQPWE6DLCMM33HXYA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530765398 (2h48m51.546434998s) [noise=true , address=77.90.31.15:4717 , piece_id=KS5ZU2QOWKQENYRO6GBTNG7MEDAADMCZQQSTP7VWER3F3ETFN6XQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 139.59.144.22:60646->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530789282 (2h48m51.546458882s) [seg_pos={0 0}, upload=2692]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:59:48.650623196 (2h48m51.666292797s) [upload=2692]: Done uploading segments... Mar 10 18:59:48.650672655 (2h48m51.666342258s) [upload=2692]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.09350407s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................X..................X......X] duration: 1m0.122834102s trace: https://jaeger.dev.storj.io/trace/542a9c4646f77fb5 FAIL (2026-03-10 18:59:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 18:58:48.813308868 (2h47m51.828978477s) [upload=2692]: Done starting upload Mar 10 18:58:48.813424821 (2h47m51.829094415s) [upload=2692]: Uploading segments... Mar 10 18:58:48.813771258 (2h47m51.829440858s) [upload=2692]: Got next segment. Inline: false Mar 10 18:58:48.813822395 (2h47m51.829491999s) [upload=2692]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:58:48.813828121 (2h47m51.829497722s) [upload=2692]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:58:48.944926562 (2h47m51.960596155s) [seg_pos={0 0}, upload=2692]: Begin upload segment... Mar 10 18:58:48.944983382 (2h47m51.960652985s) [seg_pos={0 0}, upload=2692]: Done begin upload segment. Mar 10 18:58:48.945013521 (2h47m51.960683126s) [noise=true , address=77.90.31.17:4778 , piece_id=EGXEOIXDD7UCWD5WP4GEZMIAVZYHDBZCKCRNQPWE6DLCMM33HXYA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.946083132 (2h47m51.961752742s) [noise=true , address=64.49.13.27:4957 , piece_id=2TZTPQ77F3WNPQABUTFYCOBWGNTBWY6WZO4BJSPPYGPXKNAU2Y6A, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.946361136 (2h47m51.962030726s) [noise=true , address=167.17.72.70:4773 , piece_id=VC6Z43LAQHKPRZHJOQBJDB3FGQUVWKLMNWYBDRKZ5MAYN74QI7YQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.946884834 (2h47m51.962554445s) [noise=true , address=79.127.219.46:4904 , piece_id=ICXF2DTBO4MHPIIOFCGRUX7XYXBGHKA6Y5LEOMVRTZPR5NMTKQLQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.947061928 (2h47m51.962731538s) [noise=true , address=77.90.31.21:4908 , piece_id=MGLX6W62QSSMBHIAXPWCUQ4GDSTB5GCHVC7V7DIORVD57FW6VCXQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.947372660 (2h47m51.963042268s) [noise=true , address=45.91.116.9:4274 , piece_id=5XTDYTSS4YMF5LTZXXGOTE74IP4O7QBWR3WXYPEJKCQ2N6H5P3AA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.947594049 (2h47m51.963263658s) [noise=true , address=79.127.205.235:4272 , piece_id=HKNWRACPWIEHUME6E2MNHYTZJ4UAST46SYMBYIE336BO5HZU35VA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.947864759 (2h47m51.96353437s) [noise=true , address=79.127.205.233:4144 , piece_id=2MAH5ZDWDUULY7CI2OX3WJJ4U2MQTW5U3YYW6J4HV6KC5WIQC4EQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.948080941 (2h47m51.963750552s) [noise=true , address=64.49.13.139:4478 , piece_id=BLIB3MQ2KR3I2THBDIP5CIATPDY656PPAEETACFG65T25YTOOOYA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.948313384 (2h47m51.963982991s) [noise=true , address=77.90.31.18:4829 , piece_id=MV6Y3DRAKTJM3C4WJ4XBCLECORU3JH47DWF7MUFSWFWE4EZ6JOHQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.948767373 (2h47m51.964436981s) [noise=true , address=77.90.31.15:4717 , piece_id=KS5ZU2QOWKQENYRO6GBTNG7MEDAADMCZQQSTP7VWER3F3ETFN6XQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.948911100 (2h47m51.964580712s) [noise=true , address=167.17.72.69:4716 , piece_id=53SRRSQC6RBKWWHB7LQDBIMUMO4CWX4XYOBVHNC6KVJNPA4ZWUZA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.949333055 (2h47m51.965002716s) [noise=true , address=45.91.116.7:4036 , piece_id=XDKWF3ITQFPJULUQQ5NLKMR7BIB3YFFB33U7NISIBYUMSB2E2FUA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.949421824 (2h47m51.965091438s) [noise=true , address=77.90.31.10:4541 , piece_id=A6FZGPQOZ4XGQ5PJANJ5NHXNBMZVRXZ76FW7MT77ZR55IB6HHURA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.949877571 (2h47m51.965547227s) [noise=true , address=5.231.73.3:4034 , piece_id=4FQEKWBSA3LOBC5Z7SDPAD6KYVGFXA2FLASNNWI7SVWMRP2CVLMQ, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.949954655 (2h47m51.965624263s) [noise=true , address=45.91.116.6:4002 , piece_id=7TDZ66PYN47BYXEVHI5J47OCNXVG5V44DEK4M6EEUECBUYV2RW2A, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.950451735 (2h47m51.966121344s) [noise=true , address=193.149.164.68:4068 , piece_id=5AXNZSIXZVP7QRRCNYH6WJSVFZMX4INBU4PIEMFUVD6SY6QFWDUA, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:48.950939313 (2h47m51.966608919s) [noise=true , address=167.17.72.71:4816 , piece_id=3XQB2F5SSCIPO4RV3SF2QTGEXNZWBTD3C6IRO6AMDKBHZYROSO4A, seg_pos={0 0}, upload=2692]: Uploading piece... Mar 10 18:58:49.515288752 (2h47m52.530958353s) [upload=2692]: Next returned nil segment Mar 10 18:58:49.515316263 (2h47m52.530985864s) [upload=2692]: Waiting for error group managing segments... Mar 10 18:58:49.668312744 (2h47m52.683982347s) [noise=true , address=45.91.116.7:4036 , piece_id=XDKWF3ITQFPJULUQQ5NLKMR7BIB3YFFB33U7NISIBYUMSB2E2FUA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:49.681137054 (2h47m52.696806654s) [noise=true , address=45.91.116.9:4274 , piece_id=5XTDYTSS4YMF5LTZXXGOTE74IP4O7QBWR3WXYPEJKCQ2N6H5P3AA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:49.689138627 (2h47m52.704808251s) [noise=true , address=45.91.116.6:4002 , piece_id=7TDZ66PYN47BYXEVHI5J47OCNXVG5V44DEK4M6EEUECBUYV2RW2A, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:49.717308168 (2h47m52.732977769s) [noise=true , address=79.127.219.46:4904 , piece_id=ICXF2DTBO4MHPIIOFCGRUX7XYXBGHKA6Y5LEOMVRTZPR5NMTKQLQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:49.934665513 (2h47m52.950335113s) [noise=true , address=5.231.73.3:4034 , piece_id=4FQEKWBSA3LOBC5Z7SDPAD6KYVGFXA2FLASNNWI7SVWMRP2CVLMQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.024032552 (2h47m53.039702152s) [noise=true , address=167.17.72.71:4816 , piece_id=3XQB2F5SSCIPO4RV3SF2QTGEXNZWBTD3C6IRO6AMDKBHZYROSO4A, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.041285961 (2h47m53.056955562s) [noise=true , address=167.17.72.70:4773 , piece_id=VC6Z43LAQHKPRZHJOQBJDB3FGQUVWKLMNWYBDRKZ5MAYN74QI7YQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.082585315 (2h47m53.098254916s) [noise=true , address=167.17.72.69:4716 , piece_id=53SRRSQC6RBKWWHB7LQDBIMUMO4CWX4XYOBVHNC6KVJNPA4ZWUZA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.168883082 (2h47m53.184552685s) [noise=true , address=64.49.13.27:4957 , piece_id=2TZTPQ77F3WNPQABUTFYCOBWGNTBWY6WZO4BJSPPYGPXKNAU2Y6A, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.176996442 (2h47m53.192666051s) [noise=true , address=193.149.164.68:4068 , piece_id=5AXNZSIXZVP7QRRCNYH6WJSVFZMX4INBU4PIEMFUVD6SY6QFWDUA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.205136065 (2h47m53.220805669s) [noise=true , address=64.49.13.139:4478 , piece_id=BLIB3MQ2KR3I2THBDIP5CIATPDY656PPAEETACFG65T25YTOOOYA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.236254587 (2h47m53.251924194s) [noise=true , address=79.127.205.235:4272 , piece_id=HKNWRACPWIEHUME6E2MNHYTZJ4UAST46SYMBYIE336BO5HZU35VA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:58:50.362717963 (2h47m53.37838757s) [noise=true , address=79.127.205.233:4144 , piece_id=2MAH5ZDWDUULY7CI2OX3WJJ4U2MQTW5U3YYW6J4HV6KC5WIQC4EQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: <nil> Mar 10 18:59:48.529863846 (2h48m51.54553345s) [noise=true , address=77.90.31.21:4908 , piece_id=MGLX6W62QSSMBHIAXPWCUQ4GDSTB5GCHVC7V7DIORVD57FW6VCXQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:13hCBBV3cxyTWCAuG8QGGAoJRb6S8Ay23HpXYyBgRX4HZSSpga, address:77.90.31.21:4908): protocol: rpc: dial tcp 77.90.31.21:4908: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530274966 (2h48m51.545944572s) [noise=true , address=77.90.31.18:4829 , piece_id=MV6Y3DRAKTJM3C4WJ4XBCLECORU3JH47DWF7MUFSWFWE4EZ6JOHQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, address:77.90.31.18:4829): protocol: rpc: dial tcp 77.90.31.18:4829: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530469932 (2h48m51.546139538s) [noise=true , address=77.90.31.10:4541 , piece_id=A6FZGPQOZ4XGQ5PJANJ5NHXNBMZVRXZ76FW7MT77ZR55IB6HHURA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:1277g6Fg28bEwNXpQEsMvLGSfggDDNn5wPHrkv4inaVD8j71rwW, address:77.90.31.10:4541): protocol: write tcp 139.59.144.22:47358->77.90.31.10:4541: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530721033 (2h48m51.546390642s) [noise=true , address=77.90.31.17:4778 , piece_id=EGXEOIXDD7UCWD5WP4GEZMIAVZYHDBZCKCRNQPWE6DLCMM33HXYA, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:12icjsyKdwPqDddPbEZpCJvX1pxmTLSy8ZfFUMCJU1SxPsKGdZo, address:77.90.31.17:4778): protocol: rpc: dial tcp 77.90.31.17:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530765398 (2h48m51.546434998s) [noise=true , address=77.90.31.15:4717 , piece_id=KS5ZU2QOWKQENYRO6GBTNG7MEDAADMCZQQSTP7VWER3F3ETFN6XQ, seg_pos={0 0}, upload=2692]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: write tcp 139.59.144.22:60646->77.90.31.15:4717: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:59:48.530789282 (2h48m51.546458882s) [seg_pos={0 0}, upload=2692]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:59:48.650623196 (2h48m51.666292797s) [upload=2692]: Done uploading segments... Mar 10 18:59:48.650672655 (2h48m51.666342258s) [upload=2692]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.122896853s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 1.691780051s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:59:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.691835992s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 1.750344168s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:58:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.750390875s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 3.129168825s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:58:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.129192837s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 1.652705839s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:58:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.652728105s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X..................X......] duration: 3.272916912s trace: https://jaeger.dev.storj.io/trace/45e50e99ff483a21 FAIL (2026-03-10 18:57:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:53:23.963024048 (2h42m26.978693638s) [upload=2601]: Uploading segments... Mar 10 18:53:23.963248191 (2h42m26.978917791s) [upload=2601]: Got next segment. Inline: false Mar 10 18:53:23.963287682 (2h42m26.978957283s) [upload=2601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:53:23.963294790 (2h42m26.978964391s) [upload=2601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:53:24.075742829 (2h42m27.091412429s) [seg_pos={0 0}, upload=2601]: Begin upload segment... Mar 10 18:53:24.075824274 (2h42m27.091493873s) [noise=true , address=77.90.31.16:4744 , piece_id=ZO6L74CV2BTB2BSAVYYK7DEOOBNABQMTEENGNCKOT77CEOZ4OJ2Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.076881373 (2h42m27.092550982s) [noise=true , address=145.79.227.14:4857 , piece_id=XI7HGIEGCMXKCHX6UKVXYZE3SU7XWFVMK2VKH52FLW24IL3G77XQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.076960458 (2h42m27.092630074s) [seg_pos={0 0}, upload=2601]: Done begin upload segment. Mar 10 18:53:24.077045935 (2h42m27.092715546s) [noise=true , address=193.149.164.71:4298 , piece_id=J5I4GTG3XEBLXCYF247GRY3XYF6MRAQRTKL6GRKRYICSZJKRXC2Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077324828 (2h42m27.092994432s) [noise=true , address=77.90.31.10:4543 , piece_id=PX5CLESQSWQK3AEZPJEWTLSHCVC6KCNDWBAASGZABMXF462ERBHA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077624167 (2h42m27.093293771s) [noise=true , address=79.127.205.233:4151 , piece_id=QDGRZYQH7KSXC7BDVSZ2S7PW2JAYDLFILCA5W5YJ2NQ3G4JT4IWQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077685586 (2h42m27.093355186s) [noise=true , address=193.149.164.68:4065 , piece_id=EUQWI3AF46AI5HCEMHONUMADABQTESHOJ36OUQRI7SJL22I7M5PQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078047202 (2h42m27.093716802s) [noise=true , address=145.79.227.22:4540 , piece_id=IZ5RP4Z3QI2ONMNPUDWDRLFX74TVSJRIHOXPF2ZR2J4OUMXX47QA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078328012 (2h42m27.093997612s) [noise=true , address=193.149.164.66:4969 , piece_id=MFZ6NZMIPCTGID3H5XK4BTXBPFILYULFV6T6SXZOP6ITKOLSPXBQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078568581 (2h42m27.094238182s) [noise=true , address=77.90.31.14:4678 , piece_id=Z3YFLWKDFSE7BCETV2TEJW5RR5NMUCMSF5DK25AHUDCLUZNLGRNA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078736037 (2h42m27.094405644s) [noise=true , address=79.127.205.237:4237 , piece_id=4GE2PODFMOG3MCZ52KPAEKBFWOPJZVVDKWYKPMH2O4ONNW4YHNZQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079043500 (2h42m27.094713101s) [noise=true , address=77.90.31.17:4787 , piece_id=6BJ4MUGZFKRDD2IRAQ2FKQOELDZ5N3DXJY7AXRKP6YI3PUKNG6JQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079327522 (2h42m27.094997129s) [noise=true , address=167.17.72.68:4652 , piece_id=JUTTSPJXNTMTTZYUDVBA4U2T4N72VXISVXDZI2D4CAO7XXOZ665Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079532184 (2h42m27.095201771s) [noise=true , address=77.90.31.21:4900 , piece_id=GBS52L7VJXDYCJNBS67Y5LO327OH4OUHMLBBQQZQFQ66GSMITDZA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079929000 (2h42m27.095598602s) [noise=true , address=193.149.164.67:4008 , piece_id=OUQLK2PBYXRVFSLBCNAKUTANWVS4WMLLAN63QQSINGYBXPUHNSCQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080282831 (2h42m27.09595243s) [noise=true , address=79.127.219.41:4933 , piece_id=K2HB5Z2F7MYEDXLZ7TTS2LPJYE2O6VO2PC2324WCWKUODZRUYYHQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080506232 (2h42m27.096175841s) [noise=true , address=45.91.116.6:4987 , piece_id=ZCAMZQEIZY4J7EJYPF6I4XMNHB72F55YLZ4TO2XSHGWUAD3P6GJA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080832464 (2h42m27.096502063s) [noise=true , address=45.91.116.7:4038 , piece_id=CP4BYMO6GL2KGSY4NFJ4HEXMV2Y536L2KFNJCCCSVSCMOGRTAT7Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.081072697 (2h42m27.09674236s) [noise=true , address=145.79.227.38:4011 , piece_id=GKMLBTXPSJEIOBSSLOMCG4N4LVES4HU4PUYV4MVVPFZ3NYJYGQSA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.626983481 (2h42m27.642653088s) [upload=2601]: Next returned nil segment Mar 10 18:53:24.627033969 (2h42m27.642703576s) [upload=2601]: Waiting for error group managing segments... Mar 10 18:53:24.792150361 (2h42m27.807819964s) [noise=true , address=45.91.116.6:4987 , piece_id=ZCAMZQEIZY4J7EJYPF6I4XMNHB72F55YLZ4TO2XSHGWUAD3P6GJA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.794588895 (2h42m27.810258498s) [noise=true , address=45.91.116.7:4038 , piece_id=CP4BYMO6GL2KGSY4NFJ4HEXMV2Y536L2KFNJCCCSVSCMOGRTAT7Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.803660179 (2h42m27.81932978s) [noise=true , address=145.79.227.22:4540 , piece_id=IZ5RP4Z3QI2ONMNPUDWDRLFX74TVSJRIHOXPF2ZR2J4OUMXX47QA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.804920520 (2h42m27.820590125s) [noise=true , address=145.79.227.14:4857 , piece_id=XI7HGIEGCMXKCHX6UKVXYZE3SU7XWFVMK2VKH52FLW24IL3G77XQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.851028178 (2h42m27.866697779s) [noise=true , address=79.127.219.41:4933 , piece_id=K2HB5Z2F7MYEDXLZ7TTS2LPJYE2O6VO2PC2324WCWKUODZRUYYHQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.874104656 (2h42m27.889774259s) [noise=true , address=145.79.227.38:4011 , piece_id=GKMLBTXPSJEIOBSSLOMCG4N4LVES4HU4PUYV4MVVPFZ3NYJYGQSA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.297801558 (2h42m28.313471158s) [noise=true , address=193.149.164.71:4298 , piece_id=J5I4GTG3XEBLXCYF247GRY3XYF6MRAQRTKL6GRKRYICSZJKRXC2Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.309133458 (2h42m28.324803058s) [noise=true , address=193.149.164.67:4008 , piece_id=OUQLK2PBYXRVFSLBCNAKUTANWVS4WMLLAN63QQSINGYBXPUHNSCQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.319325529 (2h42m28.33499513s) [noise=true , address=167.17.72.68:4652 , piece_id=JUTTSPJXNTMTTZYUDVBA4U2T4N72VXISVXDZI2D4CAO7XXOZ665Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.320674927 (2h42m28.336344527s) [noise=true , address=193.149.164.68:4065 , piece_id=EUQWI3AF46AI5HCEMHONUMADABQTESHOJ36OUQRI7SJL22I7M5PQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.341330476 (2h42m28.357000076s) [noise=true , address=193.149.164.66:4969 , piece_id=MFZ6NZMIPCTGID3H5XK4BTXBPFILYULFV6T6SXZOP6ITKOLSPXBQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.397697276 (2h42m28.413366879s) [noise=true , address=79.127.205.237:4237 , piece_id=4GE2PODFMOG3MCZ52KPAEKBFWOPJZVVDKWYKPMH2O4ONNW4YHNZQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.426732423 (2h42m28.442402024s) [noise=true , address=79.127.205.233:4151 , piece_id=QDGRZYQH7KSXC7BDVSZ2S7PW2JAYDLFILCA5W5YJ2NQ3G4JT4IWQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:54:23.692281805 (2h43m26.707951414s) [noise=true , address=77.90.31.21:4900 , piece_id=GBS52L7VJXDYCJNBS67Y5LO327OH4OUHMLBBQQZQFQ66GSMITDZA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 139.59.144.22:38624->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692441708 (2h43m26.708111314s) [noise=true , address=77.90.31.10:4543 , piece_id=PX5CLESQSWQK3AEZPJEWTLSHCVC6KCNDWBAASGZABMXF462ERBHA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: write tcp 139.59.144.22:42530->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692547439 (2h43m26.708217042s) [noise=true , address=77.90.31.16:4744 , piece_id=ZO6L74CV2BTB2BSAVYYK7DEOOBNABQMTEENGNCKOT77CEOZ4OJ2Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, address:77.90.31.16:4744): protocol: write tcp 139.59.144.22:49018->77.90.31.16:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692686004 (2h43m26.708355607s) [noise=true , address=77.90.31.17:4787 , piece_id=6BJ4MUGZFKRDD2IRAQ2FKQOELDZ5N3DXJY7AXRKP6YI3PUKNG6JQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: write tcp 139.59.144.22:41590->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692776030 (2h43m26.708445636s) [noise=true , address=77.90.31.14:4678 , piece_id=Z3YFLWKDFSE7BCETV2TEJW5RR5NMUCMSF5DK25AHUDCLUZNLGRNA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: rpc: dial tcp 77.90.31.14:4678: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692821193 (2h43m26.70849078s) [seg_pos={0 0}, upload=2601]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:54:23.803008979 (2h43m26.818678579s) [upload=2601]: Done uploading segments... Mar 10 18:54:23.803048244 (2h43m26.818717848s) [upload=2601]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.272942317s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.723688524s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:57:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.723716623s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 1.713265748s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:57:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.713291509s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 2.106247322s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:57:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.10633869s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 2.224745554s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:57:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.224785396s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.841412297s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:56:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.841441813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X..................X.....] duration: 3.156570419s trace: https://jaeger.dev.storj.io/trace/45e50e99ff483a21 FAIL (2026-03-10 18:56:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:53:23.963024048 (2h42m26.978693638s) [upload=2601]: Uploading segments... Mar 10 18:53:23.963248191 (2h42m26.978917791s) [upload=2601]: Got next segment. Inline: false Mar 10 18:53:23.963287682 (2h42m26.978957283s) [upload=2601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:53:23.963294790 (2h42m26.978964391s) [upload=2601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:53:24.075742829 (2h42m27.091412429s) [seg_pos={0 0}, upload=2601]: Begin upload segment... Mar 10 18:53:24.075824274 (2h42m27.091493873s) [noise=true , address=77.90.31.16:4744 , piece_id=ZO6L74CV2BTB2BSAVYYK7DEOOBNABQMTEENGNCKOT77CEOZ4OJ2Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.076881373 (2h42m27.092550982s) [noise=true , address=145.79.227.14:4857 , piece_id=XI7HGIEGCMXKCHX6UKVXYZE3SU7XWFVMK2VKH52FLW24IL3G77XQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.076960458 (2h42m27.092630074s) [seg_pos={0 0}, upload=2601]: Done begin upload segment. Mar 10 18:53:24.077045935 (2h42m27.092715546s) [noise=true , address=193.149.164.71:4298 , piece_id=J5I4GTG3XEBLXCYF247GRY3XYF6MRAQRTKL6GRKRYICSZJKRXC2Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077324828 (2h42m27.092994432s) [noise=true , address=77.90.31.10:4543 , piece_id=PX5CLESQSWQK3AEZPJEWTLSHCVC6KCNDWBAASGZABMXF462ERBHA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077624167 (2h42m27.093293771s) [noise=true , address=79.127.205.233:4151 , piece_id=QDGRZYQH7KSXC7BDVSZ2S7PW2JAYDLFILCA5W5YJ2NQ3G4JT4IWQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077685586 (2h42m27.093355186s) [noise=true , address=193.149.164.68:4065 , piece_id=EUQWI3AF46AI5HCEMHONUMADABQTESHOJ36OUQRI7SJL22I7M5PQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078047202 (2h42m27.093716802s) [noise=true , address=145.79.227.22:4540 , piece_id=IZ5RP4Z3QI2ONMNPUDWDRLFX74TVSJRIHOXPF2ZR2J4OUMXX47QA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078328012 (2h42m27.093997612s) [noise=true , address=193.149.164.66:4969 , piece_id=MFZ6NZMIPCTGID3H5XK4BTXBPFILYULFV6T6SXZOP6ITKOLSPXBQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078568581 (2h42m27.094238182s) [noise=true , address=77.90.31.14:4678 , piece_id=Z3YFLWKDFSE7BCETV2TEJW5RR5NMUCMSF5DK25AHUDCLUZNLGRNA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078736037 (2h42m27.094405644s) [noise=true , address=79.127.205.237:4237 , piece_id=4GE2PODFMOG3MCZ52KPAEKBFWOPJZVVDKWYKPMH2O4ONNW4YHNZQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079043500 (2h42m27.094713101s) [noise=true , address=77.90.31.17:4787 , piece_id=6BJ4MUGZFKRDD2IRAQ2FKQOELDZ5N3DXJY7AXRKP6YI3PUKNG6JQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079327522 (2h42m27.094997129s) [noise=true , address=167.17.72.68:4652 , piece_id=JUTTSPJXNTMTTZYUDVBA4U2T4N72VXISVXDZI2D4CAO7XXOZ665Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079532184 (2h42m27.095201771s) [noise=true , address=77.90.31.21:4900 , piece_id=GBS52L7VJXDYCJNBS67Y5LO327OH4OUHMLBBQQZQFQ66GSMITDZA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079929000 (2h42m27.095598602s) [noise=true , address=193.149.164.67:4008 , piece_id=OUQLK2PBYXRVFSLBCNAKUTANWVS4WMLLAN63QQSINGYBXPUHNSCQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080282831 (2h42m27.09595243s) [noise=true , address=79.127.219.41:4933 , piece_id=K2HB5Z2F7MYEDXLZ7TTS2LPJYE2O6VO2PC2324WCWKUODZRUYYHQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080506232 (2h42m27.096175841s) [noise=true , address=45.91.116.6:4987 , piece_id=ZCAMZQEIZY4J7EJYPF6I4XMNHB72F55YLZ4TO2XSHGWUAD3P6GJA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080832464 (2h42m27.096502063s) [noise=true , address=45.91.116.7:4038 , piece_id=CP4BYMO6GL2KGSY4NFJ4HEXMV2Y536L2KFNJCCCSVSCMOGRTAT7Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.081072697 (2h42m27.09674236s) [noise=true , address=145.79.227.38:4011 , piece_id=GKMLBTXPSJEIOBSSLOMCG4N4LVES4HU4PUYV4MVVPFZ3NYJYGQSA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.626983481 (2h42m27.642653088s) [upload=2601]: Next returned nil segment Mar 10 18:53:24.627033969 (2h42m27.642703576s) [upload=2601]: Waiting for error group managing segments... Mar 10 18:53:24.792150361 (2h42m27.807819964s) [noise=true , address=45.91.116.6:4987 , piece_id=ZCAMZQEIZY4J7EJYPF6I4XMNHB72F55YLZ4TO2XSHGWUAD3P6GJA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.794588895 (2h42m27.810258498s) [noise=true , address=45.91.116.7:4038 , piece_id=CP4BYMO6GL2KGSY4NFJ4HEXMV2Y536L2KFNJCCCSVSCMOGRTAT7Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.803660179 (2h42m27.81932978s) [noise=true , address=145.79.227.22:4540 , piece_id=IZ5RP4Z3QI2ONMNPUDWDRLFX74TVSJRIHOXPF2ZR2J4OUMXX47QA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.804920520 (2h42m27.820590125s) [noise=true , address=145.79.227.14:4857 , piece_id=XI7HGIEGCMXKCHX6UKVXYZE3SU7XWFVMK2VKH52FLW24IL3G77XQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.851028178 (2h42m27.866697779s) [noise=true , address=79.127.219.41:4933 , piece_id=K2HB5Z2F7MYEDXLZ7TTS2LPJYE2O6VO2PC2324WCWKUODZRUYYHQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.874104656 (2h42m27.889774259s) [noise=true , address=145.79.227.38:4011 , piece_id=GKMLBTXPSJEIOBSSLOMCG4N4LVES4HU4PUYV4MVVPFZ3NYJYGQSA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.297801558 (2h42m28.313471158s) [noise=true , address=193.149.164.71:4298 , piece_id=J5I4GTG3XEBLXCYF247GRY3XYF6MRAQRTKL6GRKRYICSZJKRXC2Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.309133458 (2h42m28.324803058s) [noise=true , address=193.149.164.67:4008 , piece_id=OUQLK2PBYXRVFSLBCNAKUTANWVS4WMLLAN63QQSINGYBXPUHNSCQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.319325529 (2h42m28.33499513s) [noise=true , address=167.17.72.68:4652 , piece_id=JUTTSPJXNTMTTZYUDVBA4U2T4N72VXISVXDZI2D4CAO7XXOZ665Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.320674927 (2h42m28.336344527s) [noise=true , address=193.149.164.68:4065 , piece_id=EUQWI3AF46AI5HCEMHONUMADABQTESHOJ36OUQRI7SJL22I7M5PQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.341330476 (2h42m28.357000076s) [noise=true , address=193.149.164.66:4969 , piece_id=MFZ6NZMIPCTGID3H5XK4BTXBPFILYULFV6T6SXZOP6ITKOLSPXBQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.397697276 (2h42m28.413366879s) [noise=true , address=79.127.205.237:4237 , piece_id=4GE2PODFMOG3MCZ52KPAEKBFWOPJZVVDKWYKPMH2O4ONNW4YHNZQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.426732423 (2h42m28.442402024s) [noise=true , address=79.127.205.233:4151 , piece_id=QDGRZYQH7KSXC7BDVSZ2S7PW2JAYDLFILCA5W5YJ2NQ3G4JT4IWQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:54:23.692281805 (2h43m26.707951414s) [noise=true , address=77.90.31.21:4900 , piece_id=GBS52L7VJXDYCJNBS67Y5LO327OH4OUHMLBBQQZQFQ66GSMITDZA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 139.59.144.22:38624->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692441708 (2h43m26.708111314s) [noise=true , address=77.90.31.10:4543 , piece_id=PX5CLESQSWQK3AEZPJEWTLSHCVC6KCNDWBAASGZABMXF462ERBHA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: write tcp 139.59.144.22:42530->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692547439 (2h43m26.708217042s) [noise=true , address=77.90.31.16:4744 , piece_id=ZO6L74CV2BTB2BSAVYYK7DEOOBNABQMTEENGNCKOT77CEOZ4OJ2Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, address:77.90.31.16:4744): protocol: write tcp 139.59.144.22:49018->77.90.31.16:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692686004 (2h43m26.708355607s) [noise=true , address=77.90.31.17:4787 , piece_id=6BJ4MUGZFKRDD2IRAQ2FKQOELDZ5N3DXJY7AXRKP6YI3PUKNG6JQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: write tcp 139.59.144.22:41590->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692776030 (2h43m26.708445636s) [noise=true , address=77.90.31.14:4678 , piece_id=Z3YFLWKDFSE7BCETV2TEJW5RR5NMUCMSF5DK25AHUDCLUZNLGRNA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: rpc: dial tcp 77.90.31.14:4678: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692821193 (2h43m26.70849078s) [seg_pos={0 0}, upload=2601]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:54:23.803008979 (2h43m26.818678579s) [upload=2601]: Done uploading segments... Mar 10 18:54:23.803048244 (2h43m26.818717848s) [upload=2601]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.156597129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 1.583557909s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:56:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.583584512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X..................X....] duration: 3.335221168s trace: https://jaeger.dev.storj.io/trace/45e50e99ff483a21 FAIL (2026-03-10 18:56:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:53:23.963024048 (2h42m26.978693638s) [upload=2601]: Uploading segments... Mar 10 18:53:23.963248191 (2h42m26.978917791s) [upload=2601]: Got next segment. Inline: false Mar 10 18:53:23.963287682 (2h42m26.978957283s) [upload=2601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:53:23.963294790 (2h42m26.978964391s) [upload=2601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:53:24.075742829 (2h42m27.091412429s) [seg_pos={0 0}, upload=2601]: Begin upload segment... Mar 10 18:53:24.075824274 (2h42m27.091493873s) [noise=true , address=77.90.31.16:4744 , piece_id=ZO6L74CV2BTB2BSAVYYK7DEOOBNABQMTEENGNCKOT77CEOZ4OJ2Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.076881373 (2h42m27.092550982s) [noise=true , address=145.79.227.14:4857 , piece_id=XI7HGIEGCMXKCHX6UKVXYZE3SU7XWFVMK2VKH52FLW24IL3G77XQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.076960458 (2h42m27.092630074s) [seg_pos={0 0}, upload=2601]: Done begin upload segment. Mar 10 18:53:24.077045935 (2h42m27.092715546s) [noise=true , address=193.149.164.71:4298 , piece_id=J5I4GTG3XEBLXCYF247GRY3XYF6MRAQRTKL6GRKRYICSZJKRXC2Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077324828 (2h42m27.092994432s) [noise=true , address=77.90.31.10:4543 , piece_id=PX5CLESQSWQK3AEZPJEWTLSHCVC6KCNDWBAASGZABMXF462ERBHA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077624167 (2h42m27.093293771s) [noise=true , address=79.127.205.233:4151 , piece_id=QDGRZYQH7KSXC7BDVSZ2S7PW2JAYDLFILCA5W5YJ2NQ3G4JT4IWQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077685586 (2h42m27.093355186s) [noise=true , address=193.149.164.68:4065 , piece_id=EUQWI3AF46AI5HCEMHONUMADABQTESHOJ36OUQRI7SJL22I7M5PQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078047202 (2h42m27.093716802s) [noise=true , address=145.79.227.22:4540 , piece_id=IZ5RP4Z3QI2ONMNPUDWDRLFX74TVSJRIHOXPF2ZR2J4OUMXX47QA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078328012 (2h42m27.093997612s) [noise=true , address=193.149.164.66:4969 , piece_id=MFZ6NZMIPCTGID3H5XK4BTXBPFILYULFV6T6SXZOP6ITKOLSPXBQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078568581 (2h42m27.094238182s) [noise=true , address=77.90.31.14:4678 , piece_id=Z3YFLWKDFSE7BCETV2TEJW5RR5NMUCMSF5DK25AHUDCLUZNLGRNA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078736037 (2h42m27.094405644s) [noise=true , address=79.127.205.237:4237 , piece_id=4GE2PODFMOG3MCZ52KPAEKBFWOPJZVVDKWYKPMH2O4ONNW4YHNZQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079043500 (2h42m27.094713101s) [noise=true , address=77.90.31.17:4787 , piece_id=6BJ4MUGZFKRDD2IRAQ2FKQOELDZ5N3DXJY7AXRKP6YI3PUKNG6JQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079327522 (2h42m27.094997129s) [noise=true , address=167.17.72.68:4652 , piece_id=JUTTSPJXNTMTTZYUDVBA4U2T4N72VXISVXDZI2D4CAO7XXOZ665Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079532184 (2h42m27.095201771s) [noise=true , address=77.90.31.21:4900 , piece_id=GBS52L7VJXDYCJNBS67Y5LO327OH4OUHMLBBQQZQFQ66GSMITDZA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079929000 (2h42m27.095598602s) [noise=true , address=193.149.164.67:4008 , piece_id=OUQLK2PBYXRVFSLBCNAKUTANWVS4WMLLAN63QQSINGYBXPUHNSCQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080282831 (2h42m27.09595243s) [noise=true , address=79.127.219.41:4933 , piece_id=K2HB5Z2F7MYEDXLZ7TTS2LPJYE2O6VO2PC2324WCWKUODZRUYYHQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080506232 (2h42m27.096175841s) [noise=true , address=45.91.116.6:4987 , piece_id=ZCAMZQEIZY4J7EJYPF6I4XMNHB72F55YLZ4TO2XSHGWUAD3P6GJA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080832464 (2h42m27.096502063s) [noise=true , address=45.91.116.7:4038 , piece_id=CP4BYMO6GL2KGSY4NFJ4HEXMV2Y536L2KFNJCCCSVSCMOGRTAT7Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.081072697 (2h42m27.09674236s) [noise=true , address=145.79.227.38:4011 , piece_id=GKMLBTXPSJEIOBSSLOMCG4N4LVES4HU4PUYV4MVVPFZ3NYJYGQSA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.626983481 (2h42m27.642653088s) [upload=2601]: Next returned nil segment Mar 10 18:53:24.627033969 (2h42m27.642703576s) [upload=2601]: Waiting for error group managing segments... Mar 10 18:53:24.792150361 (2h42m27.807819964s) [noise=true , address=45.91.116.6:4987 , piece_id=ZCAMZQEIZY4J7EJYPF6I4XMNHB72F55YLZ4TO2XSHGWUAD3P6GJA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.794588895 (2h42m27.810258498s) [noise=true , address=45.91.116.7:4038 , piece_id=CP4BYMO6GL2KGSY4NFJ4HEXMV2Y536L2KFNJCCCSVSCMOGRTAT7Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.803660179 (2h42m27.81932978s) [noise=true , address=145.79.227.22:4540 , piece_id=IZ5RP4Z3QI2ONMNPUDWDRLFX74TVSJRIHOXPF2ZR2J4OUMXX47QA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.804920520 (2h42m27.820590125s) [noise=true , address=145.79.227.14:4857 , piece_id=XI7HGIEGCMXKCHX6UKVXYZE3SU7XWFVMK2VKH52FLW24IL3G77XQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.851028178 (2h42m27.866697779s) [noise=true , address=79.127.219.41:4933 , piece_id=K2HB5Z2F7MYEDXLZ7TTS2LPJYE2O6VO2PC2324WCWKUODZRUYYHQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.874104656 (2h42m27.889774259s) [noise=true , address=145.79.227.38:4011 , piece_id=GKMLBTXPSJEIOBSSLOMCG4N4LVES4HU4PUYV4MVVPFZ3NYJYGQSA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.297801558 (2h42m28.313471158s) [noise=true , address=193.149.164.71:4298 , piece_id=J5I4GTG3XEBLXCYF247GRY3XYF6MRAQRTKL6GRKRYICSZJKRXC2Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.309133458 (2h42m28.324803058s) [noise=true , address=193.149.164.67:4008 , piece_id=OUQLK2PBYXRVFSLBCNAKUTANWVS4WMLLAN63QQSINGYBXPUHNSCQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.319325529 (2h42m28.33499513s) [noise=true , address=167.17.72.68:4652 , piece_id=JUTTSPJXNTMTTZYUDVBA4U2T4N72VXISVXDZI2D4CAO7XXOZ665Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.320674927 (2h42m28.336344527s) [noise=true , address=193.149.164.68:4065 , piece_id=EUQWI3AF46AI5HCEMHONUMADABQTESHOJ36OUQRI7SJL22I7M5PQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.341330476 (2h42m28.357000076s) [noise=true , address=193.149.164.66:4969 , piece_id=MFZ6NZMIPCTGID3H5XK4BTXBPFILYULFV6T6SXZOP6ITKOLSPXBQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.397697276 (2h42m28.413366879s) [noise=true , address=79.127.205.237:4237 , piece_id=4GE2PODFMOG3MCZ52KPAEKBFWOPJZVVDKWYKPMH2O4ONNW4YHNZQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.426732423 (2h42m28.442402024s) [noise=true , address=79.127.205.233:4151 , piece_id=QDGRZYQH7KSXC7BDVSZ2S7PW2JAYDLFILCA5W5YJ2NQ3G4JT4IWQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:54:23.692281805 (2h43m26.707951414s) [noise=true , address=77.90.31.21:4900 , piece_id=GBS52L7VJXDYCJNBS67Y5LO327OH4OUHMLBBQQZQFQ66GSMITDZA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 139.59.144.22:38624->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692441708 (2h43m26.708111314s) [noise=true , address=77.90.31.10:4543 , piece_id=PX5CLESQSWQK3AEZPJEWTLSHCVC6KCNDWBAASGZABMXF462ERBHA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: write tcp 139.59.144.22:42530->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692547439 (2h43m26.708217042s) [noise=true , address=77.90.31.16:4744 , piece_id=ZO6L74CV2BTB2BSAVYYK7DEOOBNABQMTEENGNCKOT77CEOZ4OJ2Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, address:77.90.31.16:4744): protocol: write tcp 139.59.144.22:49018->77.90.31.16:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692686004 (2h43m26.708355607s) [noise=true , address=77.90.31.17:4787 , piece_id=6BJ4MUGZFKRDD2IRAQ2FKQOELDZ5N3DXJY7AXRKP6YI3PUKNG6JQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: write tcp 139.59.144.22:41590->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692776030 (2h43m26.708445636s) [noise=true , address=77.90.31.14:4678 , piece_id=Z3YFLWKDFSE7BCETV2TEJW5RR5NMUCMSF5DK25AHUDCLUZNLGRNA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: rpc: dial tcp 77.90.31.14:4678: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692821193 (2h43m26.70849078s) [seg_pos={0 0}, upload=2601]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:54:23.803008979 (2h43m26.818678579s) [upload=2601]: Done uploading segments... Mar 10 18:54:23.803048244 (2h43m26.818717848s) [upload=2601]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.335249015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.902624288s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:56:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.9026806s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 1.662362541s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:55:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.662388059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X..................X...] duration: 4.606818554s trace: https://jaeger.dev.storj.io/trace/45e50e99ff483a21 FAIL (2026-03-10 18:55:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:53:23.963024048 (2h42m26.978693638s) [upload=2601]: Uploading segments... Mar 10 18:53:23.963248191 (2h42m26.978917791s) [upload=2601]: Got next segment. Inline: false Mar 10 18:53:23.963287682 (2h42m26.978957283s) [upload=2601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:53:23.963294790 (2h42m26.978964391s) [upload=2601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:53:24.075742829 (2h42m27.091412429s) [seg_pos={0 0}, upload=2601]: Begin upload segment... Mar 10 18:53:24.075824274 (2h42m27.091493873s) [noise=true , address=77.90.31.16:4744 , piece_id=ZO6L74CV2BTB2BSAVYYK7DEOOBNABQMTEENGNCKOT77CEOZ4OJ2Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.076881373 (2h42m27.092550982s) [noise=true , address=145.79.227.14:4857 , piece_id=XI7HGIEGCMXKCHX6UKVXYZE3SU7XWFVMK2VKH52FLW24IL3G77XQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.076960458 (2h42m27.092630074s) [seg_pos={0 0}, upload=2601]: Done begin upload segment. Mar 10 18:53:24.077045935 (2h42m27.092715546s) [noise=true , address=193.149.164.71:4298 , piece_id=J5I4GTG3XEBLXCYF247GRY3XYF6MRAQRTKL6GRKRYICSZJKRXC2Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077324828 (2h42m27.092994432s) [noise=true , address=77.90.31.10:4543 , piece_id=PX5CLESQSWQK3AEZPJEWTLSHCVC6KCNDWBAASGZABMXF462ERBHA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077624167 (2h42m27.093293771s) [noise=true , address=79.127.205.233:4151 , piece_id=QDGRZYQH7KSXC7BDVSZ2S7PW2JAYDLFILCA5W5YJ2NQ3G4JT4IWQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077685586 (2h42m27.093355186s) [noise=true , address=193.149.164.68:4065 , piece_id=EUQWI3AF46AI5HCEMHONUMADABQTESHOJ36OUQRI7SJL22I7M5PQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078047202 (2h42m27.093716802s) [noise=true , address=145.79.227.22:4540 , piece_id=IZ5RP4Z3QI2ONMNPUDWDRLFX74TVSJRIHOXPF2ZR2J4OUMXX47QA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078328012 (2h42m27.093997612s) [noise=true , address=193.149.164.66:4969 , piece_id=MFZ6NZMIPCTGID3H5XK4BTXBPFILYULFV6T6SXZOP6ITKOLSPXBQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078568581 (2h42m27.094238182s) [noise=true , address=77.90.31.14:4678 , piece_id=Z3YFLWKDFSE7BCETV2TEJW5RR5NMUCMSF5DK25AHUDCLUZNLGRNA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078736037 (2h42m27.094405644s) [noise=true , address=79.127.205.237:4237 , piece_id=4GE2PODFMOG3MCZ52KPAEKBFWOPJZVVDKWYKPMH2O4ONNW4YHNZQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079043500 (2h42m27.094713101s) [noise=true , address=77.90.31.17:4787 , piece_id=6BJ4MUGZFKRDD2IRAQ2FKQOELDZ5N3DXJY7AXRKP6YI3PUKNG6JQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079327522 (2h42m27.094997129s) [noise=true , address=167.17.72.68:4652 , piece_id=JUTTSPJXNTMTTZYUDVBA4U2T4N72VXISVXDZI2D4CAO7XXOZ665Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079532184 (2h42m27.095201771s) [noise=true , address=77.90.31.21:4900 , piece_id=GBS52L7VJXDYCJNBS67Y5LO327OH4OUHMLBBQQZQFQ66GSMITDZA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079929000 (2h42m27.095598602s) [noise=true , address=193.149.164.67:4008 , piece_id=OUQLK2PBYXRVFSLBCNAKUTANWVS4WMLLAN63QQSINGYBXPUHNSCQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080282831 (2h42m27.09595243s) [noise=true , address=79.127.219.41:4933 , piece_id=K2HB5Z2F7MYEDXLZ7TTS2LPJYE2O6VO2PC2324WCWKUODZRUYYHQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080506232 (2h42m27.096175841s) [noise=true , address=45.91.116.6:4987 , piece_id=ZCAMZQEIZY4J7EJYPF6I4XMNHB72F55YLZ4TO2XSHGWUAD3P6GJA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080832464 (2h42m27.096502063s) [noise=true , address=45.91.116.7:4038 , piece_id=CP4BYMO6GL2KGSY4NFJ4HEXMV2Y536L2KFNJCCCSVSCMOGRTAT7Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.081072697 (2h42m27.09674236s) [noise=true , address=145.79.227.38:4011 , piece_id=GKMLBTXPSJEIOBSSLOMCG4N4LVES4HU4PUYV4MVVPFZ3NYJYGQSA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.626983481 (2h42m27.642653088s) [upload=2601]: Next returned nil segment Mar 10 18:53:24.627033969 (2h42m27.642703576s) [upload=2601]: Waiting for error group managing segments... Mar 10 18:53:24.792150361 (2h42m27.807819964s) [noise=true , address=45.91.116.6:4987 , piece_id=ZCAMZQEIZY4J7EJYPF6I4XMNHB72F55YLZ4TO2XSHGWUAD3P6GJA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.794588895 (2h42m27.810258498s) [noise=true , address=45.91.116.7:4038 , piece_id=CP4BYMO6GL2KGSY4NFJ4HEXMV2Y536L2KFNJCCCSVSCMOGRTAT7Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.803660179 (2h42m27.81932978s) [noise=true , address=145.79.227.22:4540 , piece_id=IZ5RP4Z3QI2ONMNPUDWDRLFX74TVSJRIHOXPF2ZR2J4OUMXX47QA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.804920520 (2h42m27.820590125s) [noise=true , address=145.79.227.14:4857 , piece_id=XI7HGIEGCMXKCHX6UKVXYZE3SU7XWFVMK2VKH52FLW24IL3G77XQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.851028178 (2h42m27.866697779s) [noise=true , address=79.127.219.41:4933 , piece_id=K2HB5Z2F7MYEDXLZ7TTS2LPJYE2O6VO2PC2324WCWKUODZRUYYHQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.874104656 (2h42m27.889774259s) [noise=true , address=145.79.227.38:4011 , piece_id=GKMLBTXPSJEIOBSSLOMCG4N4LVES4HU4PUYV4MVVPFZ3NYJYGQSA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.297801558 (2h42m28.313471158s) [noise=true , address=193.149.164.71:4298 , piece_id=J5I4GTG3XEBLXCYF247GRY3XYF6MRAQRTKL6GRKRYICSZJKRXC2Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.309133458 (2h42m28.324803058s) [noise=true , address=193.149.164.67:4008 , piece_id=OUQLK2PBYXRVFSLBCNAKUTANWVS4WMLLAN63QQSINGYBXPUHNSCQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.319325529 (2h42m28.33499513s) [noise=true , address=167.17.72.68:4652 , piece_id=JUTTSPJXNTMTTZYUDVBA4U2T4N72VXISVXDZI2D4CAO7XXOZ665Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.320674927 (2h42m28.336344527s) [noise=true , address=193.149.164.68:4065 , piece_id=EUQWI3AF46AI5HCEMHONUMADABQTESHOJ36OUQRI7SJL22I7M5PQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.341330476 (2h42m28.357000076s) [noise=true , address=193.149.164.66:4969 , piece_id=MFZ6NZMIPCTGID3H5XK4BTXBPFILYULFV6T6SXZOP6ITKOLSPXBQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.397697276 (2h42m28.413366879s) [noise=true , address=79.127.205.237:4237 , piece_id=4GE2PODFMOG3MCZ52KPAEKBFWOPJZVVDKWYKPMH2O4ONNW4YHNZQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.426732423 (2h42m28.442402024s) [noise=true , address=79.127.205.233:4151 , piece_id=QDGRZYQH7KSXC7BDVSZ2S7PW2JAYDLFILCA5W5YJ2NQ3G4JT4IWQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:54:23.692281805 (2h43m26.707951414s) [noise=true , address=77.90.31.21:4900 , piece_id=GBS52L7VJXDYCJNBS67Y5LO327OH4OUHMLBBQQZQFQ66GSMITDZA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 139.59.144.22:38624->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692441708 (2h43m26.708111314s) [noise=true , address=77.90.31.10:4543 , piece_id=PX5CLESQSWQK3AEZPJEWTLSHCVC6KCNDWBAASGZABMXF462ERBHA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: write tcp 139.59.144.22:42530->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692547439 (2h43m26.708217042s) [noise=true , address=77.90.31.16:4744 , piece_id=ZO6L74CV2BTB2BSAVYYK7DEOOBNABQMTEENGNCKOT77CEOZ4OJ2Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, address:77.90.31.16:4744): protocol: write tcp 139.59.144.22:49018->77.90.31.16:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692686004 (2h43m26.708355607s) [noise=true , address=77.90.31.17:4787 , piece_id=6BJ4MUGZFKRDD2IRAQ2FKQOELDZ5N3DXJY7AXRKP6YI3PUKNG6JQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: write tcp 139.59.144.22:41590->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692776030 (2h43m26.708445636s) [noise=true , address=77.90.31.14:4678 , piece_id=Z3YFLWKDFSE7BCETV2TEJW5RR5NMUCMSF5DK25AHUDCLUZNLGRNA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: rpc: dial tcp 77.90.31.14:4678: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692821193 (2h43m26.70849078s) [seg_pos={0 0}, upload=2601]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:54:23.803008979 (2h43m26.818678579s) [upload=2601]: Done uploading segments... Mar 10 18:54:23.803048244 (2h43m26.818717848s) [upload=2601]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 4.606846314s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.958979354s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:55:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.959074311s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X..................X..] duration: 3.317822955s trace: https://jaeger.dev.storj.io/trace/45e50e99ff483a21 FAIL (2026-03-10 18:54:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:53:23.963024048 (2h42m26.978693638s) [upload=2601]: Uploading segments... Mar 10 18:53:23.963248191 (2h42m26.978917791s) [upload=2601]: Got next segment. Inline: false Mar 10 18:53:23.963287682 (2h42m26.978957283s) [upload=2601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:53:23.963294790 (2h42m26.978964391s) [upload=2601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:53:24.075742829 (2h42m27.091412429s) [seg_pos={0 0}, upload=2601]: Begin upload segment... Mar 10 18:53:24.075824274 (2h42m27.091493873s) [noise=true , address=77.90.31.16:4744 , piece_id=ZO6L74CV2BTB2BSAVYYK7DEOOBNABQMTEENGNCKOT77CEOZ4OJ2Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.076881373 (2h42m27.092550982s) [noise=true , address=145.79.227.14:4857 , piece_id=XI7HGIEGCMXKCHX6UKVXYZE3SU7XWFVMK2VKH52FLW24IL3G77XQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.076960458 (2h42m27.092630074s) [seg_pos={0 0}, upload=2601]: Done begin upload segment. Mar 10 18:53:24.077045935 (2h42m27.092715546s) [noise=true , address=193.149.164.71:4298 , piece_id=J5I4GTG3XEBLXCYF247GRY3XYF6MRAQRTKL6GRKRYICSZJKRXC2Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077324828 (2h42m27.092994432s) [noise=true , address=77.90.31.10:4543 , piece_id=PX5CLESQSWQK3AEZPJEWTLSHCVC6KCNDWBAASGZABMXF462ERBHA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077624167 (2h42m27.093293771s) [noise=true , address=79.127.205.233:4151 , piece_id=QDGRZYQH7KSXC7BDVSZ2S7PW2JAYDLFILCA5W5YJ2NQ3G4JT4IWQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077685586 (2h42m27.093355186s) [noise=true , address=193.149.164.68:4065 , piece_id=EUQWI3AF46AI5HCEMHONUMADABQTESHOJ36OUQRI7SJL22I7M5PQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078047202 (2h42m27.093716802s) [noise=true , address=145.79.227.22:4540 , piece_id=IZ5RP4Z3QI2ONMNPUDWDRLFX74TVSJRIHOXPF2ZR2J4OUMXX47QA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078328012 (2h42m27.093997612s) [noise=true , address=193.149.164.66:4969 , piece_id=MFZ6NZMIPCTGID3H5XK4BTXBPFILYULFV6T6SXZOP6ITKOLSPXBQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078568581 (2h42m27.094238182s) [noise=true , address=77.90.31.14:4678 , piece_id=Z3YFLWKDFSE7BCETV2TEJW5RR5NMUCMSF5DK25AHUDCLUZNLGRNA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078736037 (2h42m27.094405644s) [noise=true , address=79.127.205.237:4237 , piece_id=4GE2PODFMOG3MCZ52KPAEKBFWOPJZVVDKWYKPMH2O4ONNW4YHNZQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079043500 (2h42m27.094713101s) [noise=true , address=77.90.31.17:4787 , piece_id=6BJ4MUGZFKRDD2IRAQ2FKQOELDZ5N3DXJY7AXRKP6YI3PUKNG6JQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079327522 (2h42m27.094997129s) [noise=true , address=167.17.72.68:4652 , piece_id=JUTTSPJXNTMTTZYUDVBA4U2T4N72VXISVXDZI2D4CAO7XXOZ665Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079532184 (2h42m27.095201771s) [noise=true , address=77.90.31.21:4900 , piece_id=GBS52L7VJXDYCJNBS67Y5LO327OH4OUHMLBBQQZQFQ66GSMITDZA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079929000 (2h42m27.095598602s) [noise=true , address=193.149.164.67:4008 , piece_id=OUQLK2PBYXRVFSLBCNAKUTANWVS4WMLLAN63QQSINGYBXPUHNSCQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080282831 (2h42m27.09595243s) [noise=true , address=79.127.219.41:4933 , piece_id=K2HB5Z2F7MYEDXLZ7TTS2LPJYE2O6VO2PC2324WCWKUODZRUYYHQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080506232 (2h42m27.096175841s) [noise=true , address=45.91.116.6:4987 , piece_id=ZCAMZQEIZY4J7EJYPF6I4XMNHB72F55YLZ4TO2XSHGWUAD3P6GJA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080832464 (2h42m27.096502063s) [noise=true , address=45.91.116.7:4038 , piece_id=CP4BYMO6GL2KGSY4NFJ4HEXMV2Y536L2KFNJCCCSVSCMOGRTAT7Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.081072697 (2h42m27.09674236s) [noise=true , address=145.79.227.38:4011 , piece_id=GKMLBTXPSJEIOBSSLOMCG4N4LVES4HU4PUYV4MVVPFZ3NYJYGQSA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.626983481 (2h42m27.642653088s) [upload=2601]: Next returned nil segment Mar 10 18:53:24.627033969 (2h42m27.642703576s) [upload=2601]: Waiting for error group managing segments... Mar 10 18:53:24.792150361 (2h42m27.807819964s) [noise=true , address=45.91.116.6:4987 , piece_id=ZCAMZQEIZY4J7EJYPF6I4XMNHB72F55YLZ4TO2XSHGWUAD3P6GJA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.794588895 (2h42m27.810258498s) [noise=true , address=45.91.116.7:4038 , piece_id=CP4BYMO6GL2KGSY4NFJ4HEXMV2Y536L2KFNJCCCSVSCMOGRTAT7Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.803660179 (2h42m27.81932978s) [noise=true , address=145.79.227.22:4540 , piece_id=IZ5RP4Z3QI2ONMNPUDWDRLFX74TVSJRIHOXPF2ZR2J4OUMXX47QA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.804920520 (2h42m27.820590125s) [noise=true , address=145.79.227.14:4857 , piece_id=XI7HGIEGCMXKCHX6UKVXYZE3SU7XWFVMK2VKH52FLW24IL3G77XQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.851028178 (2h42m27.866697779s) [noise=true , address=79.127.219.41:4933 , piece_id=K2HB5Z2F7MYEDXLZ7TTS2LPJYE2O6VO2PC2324WCWKUODZRUYYHQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.874104656 (2h42m27.889774259s) [noise=true , address=145.79.227.38:4011 , piece_id=GKMLBTXPSJEIOBSSLOMCG4N4LVES4HU4PUYV4MVVPFZ3NYJYGQSA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.297801558 (2h42m28.313471158s) [noise=true , address=193.149.164.71:4298 , piece_id=J5I4GTG3XEBLXCYF247GRY3XYF6MRAQRTKL6GRKRYICSZJKRXC2Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.309133458 (2h42m28.324803058s) [noise=true , address=193.149.164.67:4008 , piece_id=OUQLK2PBYXRVFSLBCNAKUTANWVS4WMLLAN63QQSINGYBXPUHNSCQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.319325529 (2h42m28.33499513s) [noise=true , address=167.17.72.68:4652 , piece_id=JUTTSPJXNTMTTZYUDVBA4U2T4N72VXISVXDZI2D4CAO7XXOZ665Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.320674927 (2h42m28.336344527s) [noise=true , address=193.149.164.68:4065 , piece_id=EUQWI3AF46AI5HCEMHONUMADABQTESHOJ36OUQRI7SJL22I7M5PQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.341330476 (2h42m28.357000076s) [noise=true , address=193.149.164.66:4969 , piece_id=MFZ6NZMIPCTGID3H5XK4BTXBPFILYULFV6T6SXZOP6ITKOLSPXBQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.397697276 (2h42m28.413366879s) [noise=true , address=79.127.205.237:4237 , piece_id=4GE2PODFMOG3MCZ52KPAEKBFWOPJZVVDKWYKPMH2O4ONNW4YHNZQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.426732423 (2h42m28.442402024s) [noise=true , address=79.127.205.233:4151 , piece_id=QDGRZYQH7KSXC7BDVSZ2S7PW2JAYDLFILCA5W5YJ2NQ3G4JT4IWQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:54:23.692281805 (2h43m26.707951414s) [noise=true , address=77.90.31.21:4900 , piece_id=GBS52L7VJXDYCJNBS67Y5LO327OH4OUHMLBBQQZQFQ66GSMITDZA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 139.59.144.22:38624->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692441708 (2h43m26.708111314s) [noise=true , address=77.90.31.10:4543 , piece_id=PX5CLESQSWQK3AEZPJEWTLSHCVC6KCNDWBAASGZABMXF462ERBHA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: write tcp 139.59.144.22:42530->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692547439 (2h43m26.708217042s) [noise=true , address=77.90.31.16:4744 , piece_id=ZO6L74CV2BTB2BSAVYYK7DEOOBNABQMTEENGNCKOT77CEOZ4OJ2Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, address:77.90.31.16:4744): protocol: write tcp 139.59.144.22:49018->77.90.31.16:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692686004 (2h43m26.708355607s) [noise=true , address=77.90.31.17:4787 , piece_id=6BJ4MUGZFKRDD2IRAQ2FKQOELDZ5N3DXJY7AXRKP6YI3PUKNG6JQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: write tcp 139.59.144.22:41590->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692776030 (2h43m26.708445636s) [noise=true , address=77.90.31.14:4678 , piece_id=Z3YFLWKDFSE7BCETV2TEJW5RR5NMUCMSF5DK25AHUDCLUZNLGRNA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: rpc: dial tcp 77.90.31.14:4678: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692821193 (2h43m26.70849078s) [seg_pos={0 0}, upload=2601]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:54:23.803008979 (2h43m26.818678579s) [upload=2601]: Done uploading segments... Mar 10 18:54:23.803048244 (2h43m26.818717848s) [upload=2601]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.317861108s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 1.776946165s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:54:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.77697297s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.748358212s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:54:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.748389749s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X..................X.] duration: 3.19240075s trace: https://jaeger.dev.storj.io/trace/45e50e99ff483a21 FAIL (2026-03-10 18:54:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:53:23.963024048 (2h42m26.978693638s) [upload=2601]: Uploading segments... Mar 10 18:53:23.963248191 (2h42m26.978917791s) [upload=2601]: Got next segment. Inline: false Mar 10 18:53:23.963287682 (2h42m26.978957283s) [upload=2601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:53:23.963294790 (2h42m26.978964391s) [upload=2601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:53:24.075742829 (2h42m27.091412429s) [seg_pos={0 0}, upload=2601]: Begin upload segment... Mar 10 18:53:24.075824274 (2h42m27.091493873s) [noise=true , address=77.90.31.16:4744 , piece_id=ZO6L74CV2BTB2BSAVYYK7DEOOBNABQMTEENGNCKOT77CEOZ4OJ2Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.076881373 (2h42m27.092550982s) [noise=true , address=145.79.227.14:4857 , piece_id=XI7HGIEGCMXKCHX6UKVXYZE3SU7XWFVMK2VKH52FLW24IL3G77XQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.076960458 (2h42m27.092630074s) [seg_pos={0 0}, upload=2601]: Done begin upload segment. Mar 10 18:53:24.077045935 (2h42m27.092715546s) [noise=true , address=193.149.164.71:4298 , piece_id=J5I4GTG3XEBLXCYF247GRY3XYF6MRAQRTKL6GRKRYICSZJKRXC2Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077324828 (2h42m27.092994432s) [noise=true , address=77.90.31.10:4543 , piece_id=PX5CLESQSWQK3AEZPJEWTLSHCVC6KCNDWBAASGZABMXF462ERBHA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077624167 (2h42m27.093293771s) [noise=true , address=79.127.205.233:4151 , piece_id=QDGRZYQH7KSXC7BDVSZ2S7PW2JAYDLFILCA5W5YJ2NQ3G4JT4IWQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077685586 (2h42m27.093355186s) [noise=true , address=193.149.164.68:4065 , piece_id=EUQWI3AF46AI5HCEMHONUMADABQTESHOJ36OUQRI7SJL22I7M5PQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078047202 (2h42m27.093716802s) [noise=true , address=145.79.227.22:4540 , piece_id=IZ5RP4Z3QI2ONMNPUDWDRLFX74TVSJRIHOXPF2ZR2J4OUMXX47QA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078328012 (2h42m27.093997612s) [noise=true , address=193.149.164.66:4969 , piece_id=MFZ6NZMIPCTGID3H5XK4BTXBPFILYULFV6T6SXZOP6ITKOLSPXBQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078568581 (2h42m27.094238182s) [noise=true , address=77.90.31.14:4678 , piece_id=Z3YFLWKDFSE7BCETV2TEJW5RR5NMUCMSF5DK25AHUDCLUZNLGRNA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078736037 (2h42m27.094405644s) [noise=true , address=79.127.205.237:4237 , piece_id=4GE2PODFMOG3MCZ52KPAEKBFWOPJZVVDKWYKPMH2O4ONNW4YHNZQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079043500 (2h42m27.094713101s) [noise=true , address=77.90.31.17:4787 , piece_id=6BJ4MUGZFKRDD2IRAQ2FKQOELDZ5N3DXJY7AXRKP6YI3PUKNG6JQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079327522 (2h42m27.094997129s) [noise=true , address=167.17.72.68:4652 , piece_id=JUTTSPJXNTMTTZYUDVBA4U2T4N72VXISVXDZI2D4CAO7XXOZ665Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079532184 (2h42m27.095201771s) [noise=true , address=77.90.31.21:4900 , piece_id=GBS52L7VJXDYCJNBS67Y5LO327OH4OUHMLBBQQZQFQ66GSMITDZA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079929000 (2h42m27.095598602s) [noise=true , address=193.149.164.67:4008 , piece_id=OUQLK2PBYXRVFSLBCNAKUTANWVS4WMLLAN63QQSINGYBXPUHNSCQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080282831 (2h42m27.09595243s) [noise=true , address=79.127.219.41:4933 , piece_id=K2HB5Z2F7MYEDXLZ7TTS2LPJYE2O6VO2PC2324WCWKUODZRUYYHQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080506232 (2h42m27.096175841s) [noise=true , address=45.91.116.6:4987 , piece_id=ZCAMZQEIZY4J7EJYPF6I4XMNHB72F55YLZ4TO2XSHGWUAD3P6GJA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080832464 (2h42m27.096502063s) [noise=true , address=45.91.116.7:4038 , piece_id=CP4BYMO6GL2KGSY4NFJ4HEXMV2Y536L2KFNJCCCSVSCMOGRTAT7Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.081072697 (2h42m27.09674236s) [noise=true , address=145.79.227.38:4011 , piece_id=GKMLBTXPSJEIOBSSLOMCG4N4LVES4HU4PUYV4MVVPFZ3NYJYGQSA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.626983481 (2h42m27.642653088s) [upload=2601]: Next returned nil segment Mar 10 18:53:24.627033969 (2h42m27.642703576s) [upload=2601]: Waiting for error group managing segments... Mar 10 18:53:24.792150361 (2h42m27.807819964s) [noise=true , address=45.91.116.6:4987 , piece_id=ZCAMZQEIZY4J7EJYPF6I4XMNHB72F55YLZ4TO2XSHGWUAD3P6GJA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.794588895 (2h42m27.810258498s) [noise=true , address=45.91.116.7:4038 , piece_id=CP4BYMO6GL2KGSY4NFJ4HEXMV2Y536L2KFNJCCCSVSCMOGRTAT7Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.803660179 (2h42m27.81932978s) [noise=true , address=145.79.227.22:4540 , piece_id=IZ5RP4Z3QI2ONMNPUDWDRLFX74TVSJRIHOXPF2ZR2J4OUMXX47QA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.804920520 (2h42m27.820590125s) [noise=true , address=145.79.227.14:4857 , piece_id=XI7HGIEGCMXKCHX6UKVXYZE3SU7XWFVMK2VKH52FLW24IL3G77XQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.851028178 (2h42m27.866697779s) [noise=true , address=79.127.219.41:4933 , piece_id=K2HB5Z2F7MYEDXLZ7TTS2LPJYE2O6VO2PC2324WCWKUODZRUYYHQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.874104656 (2h42m27.889774259s) [noise=true , address=145.79.227.38:4011 , piece_id=GKMLBTXPSJEIOBSSLOMCG4N4LVES4HU4PUYV4MVVPFZ3NYJYGQSA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.297801558 (2h42m28.313471158s) [noise=true , address=193.149.164.71:4298 , piece_id=J5I4GTG3XEBLXCYF247GRY3XYF6MRAQRTKL6GRKRYICSZJKRXC2Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.309133458 (2h42m28.324803058s) [noise=true , address=193.149.164.67:4008 , piece_id=OUQLK2PBYXRVFSLBCNAKUTANWVS4WMLLAN63QQSINGYBXPUHNSCQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.319325529 (2h42m28.33499513s) [noise=true , address=167.17.72.68:4652 , piece_id=JUTTSPJXNTMTTZYUDVBA4U2T4N72VXISVXDZI2D4CAO7XXOZ665Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.320674927 (2h42m28.336344527s) [noise=true , address=193.149.164.68:4065 , piece_id=EUQWI3AF46AI5HCEMHONUMADABQTESHOJ36OUQRI7SJL22I7M5PQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.341330476 (2h42m28.357000076s) [noise=true , address=193.149.164.66:4969 , piece_id=MFZ6NZMIPCTGID3H5XK4BTXBPFILYULFV6T6SXZOP6ITKOLSPXBQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.397697276 (2h42m28.413366879s) [noise=true , address=79.127.205.237:4237 , piece_id=4GE2PODFMOG3MCZ52KPAEKBFWOPJZVVDKWYKPMH2O4ONNW4YHNZQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.426732423 (2h42m28.442402024s) [noise=true , address=79.127.205.233:4151 , piece_id=QDGRZYQH7KSXC7BDVSZ2S7PW2JAYDLFILCA5W5YJ2NQ3G4JT4IWQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:54:23.692281805 (2h43m26.707951414s) [noise=true , address=77.90.31.21:4900 , piece_id=GBS52L7VJXDYCJNBS67Y5LO327OH4OUHMLBBQQZQFQ66GSMITDZA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 139.59.144.22:38624->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692441708 (2h43m26.708111314s) [noise=true , address=77.90.31.10:4543 , piece_id=PX5CLESQSWQK3AEZPJEWTLSHCVC6KCNDWBAASGZABMXF462ERBHA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: write tcp 139.59.144.22:42530->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692547439 (2h43m26.708217042s) [noise=true , address=77.90.31.16:4744 , piece_id=ZO6L74CV2BTB2BSAVYYK7DEOOBNABQMTEENGNCKOT77CEOZ4OJ2Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, address:77.90.31.16:4744): protocol: write tcp 139.59.144.22:49018->77.90.31.16:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692686004 (2h43m26.708355607s) [noise=true , address=77.90.31.17:4787 , piece_id=6BJ4MUGZFKRDD2IRAQ2FKQOELDZ5N3DXJY7AXRKP6YI3PUKNG6JQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: write tcp 139.59.144.22:41590->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692776030 (2h43m26.708445636s) [noise=true , address=77.90.31.14:4678 , piece_id=Z3YFLWKDFSE7BCETV2TEJW5RR5NMUCMSF5DK25AHUDCLUZNLGRNA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: rpc: dial tcp 77.90.31.14:4678: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692821193 (2h43m26.70849078s) [seg_pos={0 0}, upload=2601]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:54:23.803008979 (2h43m26.818678579s) [upload=2601]: Done uploading segments... Mar 10 18:54:23.803048244 (2h43m26.818717848s) [upload=2601]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 3.19241677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X..................X] duration: 1m0.113214242s trace: https://jaeger.dev.storj.io/trace/45e50e99ff483a21 FAIL (2026-03-10 18:54:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Mar 10 18:53:23.963024048 (2h42m26.978693638s) [upload=2601]: Uploading segments... Mar 10 18:53:23.963248191 (2h42m26.978917791s) [upload=2601]: Got next segment. Inline: false Mar 10 18:53:23.963287682 (2h42m26.978957283s) [upload=2601]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:53:23.963294790 (2h42m26.978964391s) [upload=2601]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:53:24.075742829 (2h42m27.091412429s) [seg_pos={0 0}, upload=2601]: Begin upload segment... Mar 10 18:53:24.075824274 (2h42m27.091493873s) [noise=true , address=77.90.31.16:4744 , piece_id=ZO6L74CV2BTB2BSAVYYK7DEOOBNABQMTEENGNCKOT77CEOZ4OJ2Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.076881373 (2h42m27.092550982s) [noise=true , address=145.79.227.14:4857 , piece_id=XI7HGIEGCMXKCHX6UKVXYZE3SU7XWFVMK2VKH52FLW24IL3G77XQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.076960458 (2h42m27.092630074s) [seg_pos={0 0}, upload=2601]: Done begin upload segment. Mar 10 18:53:24.077045935 (2h42m27.092715546s) [noise=true , address=193.149.164.71:4298 , piece_id=J5I4GTG3XEBLXCYF247GRY3XYF6MRAQRTKL6GRKRYICSZJKRXC2Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077324828 (2h42m27.092994432s) [noise=true , address=77.90.31.10:4543 , piece_id=PX5CLESQSWQK3AEZPJEWTLSHCVC6KCNDWBAASGZABMXF462ERBHA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077624167 (2h42m27.093293771s) [noise=true , address=79.127.205.233:4151 , piece_id=QDGRZYQH7KSXC7BDVSZ2S7PW2JAYDLFILCA5W5YJ2NQ3G4JT4IWQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.077685586 (2h42m27.093355186s) [noise=true , address=193.149.164.68:4065 , piece_id=EUQWI3AF46AI5HCEMHONUMADABQTESHOJ36OUQRI7SJL22I7M5PQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078047202 (2h42m27.093716802s) [noise=true , address=145.79.227.22:4540 , piece_id=IZ5RP4Z3QI2ONMNPUDWDRLFX74TVSJRIHOXPF2ZR2J4OUMXX47QA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078328012 (2h42m27.093997612s) [noise=true , address=193.149.164.66:4969 , piece_id=MFZ6NZMIPCTGID3H5XK4BTXBPFILYULFV6T6SXZOP6ITKOLSPXBQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078568581 (2h42m27.094238182s) [noise=true , address=77.90.31.14:4678 , piece_id=Z3YFLWKDFSE7BCETV2TEJW5RR5NMUCMSF5DK25AHUDCLUZNLGRNA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.078736037 (2h42m27.094405644s) [noise=true , address=79.127.205.237:4237 , piece_id=4GE2PODFMOG3MCZ52KPAEKBFWOPJZVVDKWYKPMH2O4ONNW4YHNZQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079043500 (2h42m27.094713101s) [noise=true , address=77.90.31.17:4787 , piece_id=6BJ4MUGZFKRDD2IRAQ2FKQOELDZ5N3DXJY7AXRKP6YI3PUKNG6JQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079327522 (2h42m27.094997129s) [noise=true , address=167.17.72.68:4652 , piece_id=JUTTSPJXNTMTTZYUDVBA4U2T4N72VXISVXDZI2D4CAO7XXOZ665Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079532184 (2h42m27.095201771s) [noise=true , address=77.90.31.21:4900 , piece_id=GBS52L7VJXDYCJNBS67Y5LO327OH4OUHMLBBQQZQFQ66GSMITDZA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.079929000 (2h42m27.095598602s) [noise=true , address=193.149.164.67:4008 , piece_id=OUQLK2PBYXRVFSLBCNAKUTANWVS4WMLLAN63QQSINGYBXPUHNSCQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080282831 (2h42m27.09595243s) [noise=true , address=79.127.219.41:4933 , piece_id=K2HB5Z2F7MYEDXLZ7TTS2LPJYE2O6VO2PC2324WCWKUODZRUYYHQ, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080506232 (2h42m27.096175841s) [noise=true , address=45.91.116.6:4987 , piece_id=ZCAMZQEIZY4J7EJYPF6I4XMNHB72F55YLZ4TO2XSHGWUAD3P6GJA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.080832464 (2h42m27.096502063s) [noise=true , address=45.91.116.7:4038 , piece_id=CP4BYMO6GL2KGSY4NFJ4HEXMV2Y536L2KFNJCCCSVSCMOGRTAT7Q, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.081072697 (2h42m27.09674236s) [noise=true , address=145.79.227.38:4011 , piece_id=GKMLBTXPSJEIOBSSLOMCG4N4LVES4HU4PUYV4MVVPFZ3NYJYGQSA, seg_pos={0 0}, upload=2601]: Uploading piece... Mar 10 18:53:24.626983481 (2h42m27.642653088s) [upload=2601]: Next returned nil segment Mar 10 18:53:24.627033969 (2h42m27.642703576s) [upload=2601]: Waiting for error group managing segments... Mar 10 18:53:24.792150361 (2h42m27.807819964s) [noise=true , address=45.91.116.6:4987 , piece_id=ZCAMZQEIZY4J7EJYPF6I4XMNHB72F55YLZ4TO2XSHGWUAD3P6GJA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.794588895 (2h42m27.810258498s) [noise=true , address=45.91.116.7:4038 , piece_id=CP4BYMO6GL2KGSY4NFJ4HEXMV2Y536L2KFNJCCCSVSCMOGRTAT7Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.803660179 (2h42m27.81932978s) [noise=true , address=145.79.227.22:4540 , piece_id=IZ5RP4Z3QI2ONMNPUDWDRLFX74TVSJRIHOXPF2ZR2J4OUMXX47QA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.804920520 (2h42m27.820590125s) [noise=true , address=145.79.227.14:4857 , piece_id=XI7HGIEGCMXKCHX6UKVXYZE3SU7XWFVMK2VKH52FLW24IL3G77XQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.851028178 (2h42m27.866697779s) [noise=true , address=79.127.219.41:4933 , piece_id=K2HB5Z2F7MYEDXLZ7TTS2LPJYE2O6VO2PC2324WCWKUODZRUYYHQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:24.874104656 (2h42m27.889774259s) [noise=true , address=145.79.227.38:4011 , piece_id=GKMLBTXPSJEIOBSSLOMCG4N4LVES4HU4PUYV4MVVPFZ3NYJYGQSA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.297801558 (2h42m28.313471158s) [noise=true , address=193.149.164.71:4298 , piece_id=J5I4GTG3XEBLXCYF247GRY3XYF6MRAQRTKL6GRKRYICSZJKRXC2Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.309133458 (2h42m28.324803058s) [noise=true , address=193.149.164.67:4008 , piece_id=OUQLK2PBYXRVFSLBCNAKUTANWVS4WMLLAN63QQSINGYBXPUHNSCQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.319325529 (2h42m28.33499513s) [noise=true , address=167.17.72.68:4652 , piece_id=JUTTSPJXNTMTTZYUDVBA4U2T4N72VXISVXDZI2D4CAO7XXOZ665Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.320674927 (2h42m28.336344527s) [noise=true , address=193.149.164.68:4065 , piece_id=EUQWI3AF46AI5HCEMHONUMADABQTESHOJ36OUQRI7SJL22I7M5PQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.341330476 (2h42m28.357000076s) [noise=true , address=193.149.164.66:4969 , piece_id=MFZ6NZMIPCTGID3H5XK4BTXBPFILYULFV6T6SXZOP6ITKOLSPXBQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.397697276 (2h42m28.413366879s) [noise=true , address=79.127.205.237:4237 , piece_id=4GE2PODFMOG3MCZ52KPAEKBFWOPJZVVDKWYKPMH2O4ONNW4YHNZQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:53:25.426732423 (2h42m28.442402024s) [noise=true , address=79.127.205.233:4151 , piece_id=QDGRZYQH7KSXC7BDVSZ2S7PW2JAYDLFILCA5W5YJ2NQ3G4JT4IWQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: <nil> Mar 10 18:54:23.692281805 (2h43m26.707951414s) [noise=true , address=77.90.31.21:4900 , piece_id=GBS52L7VJXDYCJNBS67Y5LO327OH4OUHMLBBQQZQFQ66GSMITDZA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:129YE2Wd6LmDABJFaGVtYDwCHmStVDnPRCagD2cKPWA5PDTtZyh, address:77.90.31.21:4900): protocol: write tcp 139.59.144.22:38624->77.90.31.21:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692441708 (2h43m26.708111314s) [noise=true , address=77.90.31.10:4543 , piece_id=PX5CLESQSWQK3AEZPJEWTLSHCVC6KCNDWBAASGZABMXF462ERBHA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12hFnVoQD97rytjEVNtPo8QumC37Eq1RM9MeXBP6Bap9rhK73bX, address:77.90.31.10:4543): protocol: write tcp 139.59.144.22:42530->77.90.31.10:4543: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692547439 (2h43m26.708217042s) [noise=true , address=77.90.31.16:4744 , piece_id=ZO6L74CV2BTB2BSAVYYK7DEOOBNABQMTEENGNCKOT77CEOZ4OJ2Q, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12WXefeowedmPCcPp2FkKRbztNyXK4K6aLNNPqGTZG9NijBP4Wd, address:77.90.31.16:4744): protocol: write tcp 139.59.144.22:49018->77.90.31.16:4744: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692686004 (2h43m26.708355607s) [noise=true , address=77.90.31.17:4787 , piece_id=6BJ4MUGZFKRDD2IRAQ2FKQOELDZ5N3DXJY7AXRKP6YI3PUKNG6JQ, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: write tcp 139.59.144.22:41590->77.90.31.17:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692776030 (2h43m26.708445636s) [noise=true , address=77.90.31.14:4678 , piece_id=Z3YFLWKDFSE7BCETV2TEJW5RR5NMUCMSF5DK25AHUDCLUZNLGRNA, seg_pos={0 0}, upload=2601]: Done uploading piece. err: ecclient: upload failed (node:12beu3SUZMPNWfbeHV4tLtx9UVZoEHh4Kt7oBhT59w3bGBE1ovP, address:77.90.31.14:4678): protocol: rpc: dial tcp 77.90.31.14:4678: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:54:23.692821193 (2h43m26.70849078s) [seg_pos={0 0}, upload=2601]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Mar 10 18:54:23.803008979 (2h43m26.818678579s) [upload=2601]: Done uploading segments... Mar 10 18:54:23.803048244 (2h43m26.818717848s) [upload=2601]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.11323631s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 1.701596451s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:54:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.701622399s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 3.225136742s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:53:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.225158717s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 2.006990441s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:53:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.007015739s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.646684121s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:52:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.646710284s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 2.124496397s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:52:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.124602216s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 3.217590669s trace: https://jaeger.dev.storj.io/trace/b68ffcae175905a FAIL (2026-03-10 18:52:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05104 (2h26m26.456874693s) [upload=2343]: Got next segment. Inline: false Mar 10 18:37:23.441245955 (2h26m26.456915545s) [upload=2343]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 10 18:37:23.441251057 (2h26m26.45692066s) [upload=2343]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 10 18:37:23.557618294 (2h26m26.573287881s) [seg_pos={0 0}, upload=2343]: Begin upload segment... Mar 10 18:37:23.557695669 (2h26m26.573365278s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.558906475 (2h26m26.574576087s) [seg_pos={0 0}, upload=2343]: Done begin upload segment. Mar 10 18:37:23.558987696 (2h26m26.57465731s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.559601912 (2h26m26.575271515s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.560517706 (2h26m26.57618731s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.561695181 (2h26m26.577364785s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562028023 (2h26m26.57769763s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.562622388 (2h26m26.578292229s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564570119 (2h26m26.580239738s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.564708737 (2h26m26.58037834s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.566432612 (2h26m26.582102222s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567575901 (2h26m26.583245509s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.567768288 (2h26m26.583437899s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.568437376 (2h26m26.584106976s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.569626647 (2h26m26.585296258s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.570719385 (2h26m26.586388996s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.572104408 (2h26m26.587774016s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.573696158 (2h26m26.589365758s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:23.575002207 (2h26m26.590671812s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Uploading piece... Mar 10 18:37:24.121673515 (2h26m27.137343125s) [upload=2343]: Next returned nil segment Mar 10 18:37:24.121693564 (2h26m27.137363175s) [upload=2343]: Waiting for error group managing segments... Mar 10 18:37:24.258918473 (2h26m27.274588073s) [noise=true , address=45.91.116.4:4943 , piece_id=EOL7JSKHMYZQPZVM3Q5GN2HJC65YKJWG3KTJMICUA4PLSIERLDZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.259101868 (2h26m27.274771472s) [noise=true , address=79.127.219.41:4947 , piece_id=NI3ZGQNFOLUIRH3B4QB3FDXFLVNCYKGX24U4HROKCESWMXNKC54A, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.265791082 (2h26m27.281460686s) [noise=true , address=145.79.227.17:4000 , piece_id=AU4UBLJAXLH7JNNYKC3YB3UIJAAEWD2Q6S77CG3J6G5WJIWPDXAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.277538368 (2h26m27.293207972s) [noise=true , address=45.91.116.13:4167 , piece_id=3GJVLC4HXSBZ6D3FNOYONOTF6ZN2YUKPI3N4XFX34E7TTT7KE77Q, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.283507633 (2h26m27.299177234s) [noise=true , address=45.91.116.10:4093 , piece_id=HUH66QBVZCXFN7BLJRADGU2H42MKXL3WIICJI6LAWQF2J3YXIGRQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.288254595 (2h26m27.303924202s) [noise=true , address=45.91.116.6:4999 , piece_id=DJDQ6BIBJESD4OKWTL6JIACEQ5LU4LROC3WNXEF6APEM734BOXZQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.650345388 (2h26m27.666014988s) [noise=true , address=167.17.72.68:4659 , piece_id=4LQBLKHK5RRKVVYFSPSMDWHWG2QPDHZNG2REAVL2PMWO3VZ36NAQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.658677204 (2h26m27.674346807s) [noise=true , address=167.17.72.69:4727 , piece_id=2PMBDX2OJJD33H3GCSQRJAPZMOV5II3WLZOY5YF4HKE7GYBQU6AQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.804520312 (2h26m27.820189912s) [noise=true , address=193.149.164.69:4081 , piece_id=6DNISIGMWMIMA6OICHGJUJOR6NNMXSCJI6B4GO6VKT4MCTMKG3HA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.805528476 (2h26m27.821198079s) [noise=true , address=193.149.164.68:4068 , piece_id=7WA3675HW4UPV7ZTNJEZIJDB7IIHFWLVHTSXOR5UMLQIXNHE7AXA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.810879435 (2h26m27.826549059s) [noise=true , address=193.149.164.67:4016 , piece_id=CITHWRKZJJGV4X3VG7CPQPWX2FPJDXU4QMSFMVE6ANNKAIICEUMA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:37:24.813793209 (2h26m27.829462813s) [noise=true , address=193.149.164.66:4978 , piece_id=UFBW2KYD7LUKXGBAOB4JOTNEKUE77KZSKFFJ3EFDGHPC7QKPAFXQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: <nil> Mar 10 18:38:23.155284138 (2h27m26.170953744s) [noise=true , address=77.90.31.12:4614 , piece_id=MLXXO4453CEWAG3G5WZZLAAX7LWUSFVVWQ3FXFKODALSAWRHLEFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12wKLBwTAdUVAQmQGcafdpUcjiDE89jxdptEwXoAvV6WXf2Xbc3, address:77.90.31.12:4614): protocol: rpc: dial tcp 77.90.31.12:4614: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155359187 (2h27m26.171028792s) [noise=true , address=77.90.31.17:4796 , piece_id=FZMZYUGLKY5FPBA3GEF7UVM5IBNO2KRUSEQCJSUVL2PRRN5AWOMQ, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1KK6y5NBuZovne8n3gMvAyDcZFasZxDzorvPeHe7CefoFs27UF, address:77.90.31.17:4796): protocol: write tcp 139.59.144.22:52670->77.90.31.17:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155422072 (2h27m26.171091682s) [noise=true , address=77.90.31.15:4717 , piece_id=MLYGZT47MTTYGBLNTPA7DPM7NE6EQ6MPZYFW7KVDLFXR4MD2ORPA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc, address:77.90.31.15:4717): protocol: rpc: dial tcp 77.90.31.15:4717: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155489065 (2h27m26.171158674s) [noise=true , address=77.90.31.13:4645 , piece_id=UXFW5W5TPZBMH6IC7AT5GN2ZDMZTNDJLFH7OMNKWN5JXKAT76SFA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1b2aaYQ4o7X5yLcXG3moJ3d4oxFoNSCwsqacwg9mshJ4MPoGdg, address:77.90.31.13:4645): protocol: rpc: dial tcp 77.90.31.13:4645: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155521077 (2h27m26.171190679s) [noise=true , address=77.90.31.14:4673 , piece_id=XY5KYY3SODVHKX2EM2NJT6FRSDGSKNZFSLOKPP3T4VKC423TMQUA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:1xMwYmwF8fukP4nNY3GNLdZZ3a4YqVr35poQf3Trq7AYUNF7sh, address:77.90.31.14:4673): protocol: rpc: dial tcp 77.90.31.14:4673: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155568492 (2h27m26.171238098s) [noise=true , address=77.90.31.19:4874 , piece_id=DC4C4HMX2TD6FTW5RPS3O55IHEQ6E7OR6VKKREVKVW7FAFAPZ2XA, seg_pos={0 0}, upload=2343]: Done uploading piece. err: ecclient: upload failed (node:14J8ruKPB6P9YBLPhHpCrcLJt3WBYwNo3fGBeJ2Pdp6JfWs3W4, address:77.90.31.19:4874): protocol: rpc: dial tcp 77.90.31.19:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Mar 10 18:38:23.155592304 (2h27m26.171261905s) [seg_pos={0 0}, upload=2343]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Mar 10 18:38:23.260013959 (2h27m26.275683549s) [upload=2343]: Done uploading segments... Mar 10 18:38:23.260057302 (2h27m26.275726903s) [upload=2343]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 3.217618742s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 2.29321787s trace: https://jaeger.dev.storj.io/trace/7210fe2bf339936d FAIL (2026-03-10 18:52:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.293234089s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.700045106s trace: https://jaeger.dev.storj.io/trace/43a7d841b992769c FAIL (2026-03-10 18:51:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.700069491s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)