Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.86958153s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:45:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.869609253s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 2.503901889s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:44:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.5039147s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.XX.........................................................] duration: 2.56170847s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:43:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.561731802s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 2.39442267s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:43:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.394449352s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 2.65722065s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:42:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.657251801s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 2.554571898s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:42:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.554778493s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 2.363019289s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:41:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.363033944s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.....XX....................................................] duration: 2.629618274s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:40:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.629661834s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.....XX...................................................] duration: 2.454732509s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:40:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.455160862s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.511091739s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:39:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.511174957s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.X.....XX..................................................] duration: 2.604244634s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:39:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.604274738s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 1.546271161s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:38:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.546295015s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.388013702s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:38:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.38804432s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X.X.....XX.................................................] duration: 2.412060214s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:37:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.412087015s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 1.662454873s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:37:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.66248766s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.404721786s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:37:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.40474788s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X.X.....XX................................................] duration: 2.748913801s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:37:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.748947796s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.285000249s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:37:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.285186532s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 1.674104467s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:37:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.674137023s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.406646817s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:37:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.406691551s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.450651546s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:36:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.450678514s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X.X.....XX...............................................] duration: 2.560859708s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:36:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.560888007s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.429336223s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:36:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.429378535s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 1.716870374s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:36:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.716898211s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X.X.....XX..............................................] duration: 2.459436547s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:35:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.459526114s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X.X.....XX.............................................] duration: 2.390033366s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:35:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.390075093s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 1.582307024s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:35:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.582340612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.436155457s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:34:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.436185888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.349491047s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:34:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.349531835s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X.X.....XX............................................] duration: 2.713604418s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:34:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.713634935s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.43132866s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:34:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.431357039s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 1.562613478s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:33:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.563030537s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X.X.....XX...........................................] duration: 2.516709173s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:33:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.516735548s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 1.658160101s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:33:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.658208095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.297072268s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:32:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.297101482s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 1.608778817s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:32:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.608826937s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.472584955s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:32:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.472618175s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.X.....XX..........................................] duration: 2.669403804s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:32:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.669428581s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X........X..................................................] duration: 1.656840902s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:31:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.656863144s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.X.....XX.........................................] duration: 2.384700565s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:31:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.384728862s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.468460805s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:31:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.468489988s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.422064223s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:31:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.422086593s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X........X.................................................] duration: 1.622919228s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:30:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.622982253s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.X.....XX........................................] duration: 2.411592891s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:30:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.411625247s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.376082231s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:30:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.37611091s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.X.....XX.......................................] duration: 2.504547064s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:29:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.5045824s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X.X........X................................................] duration: 1.508546122s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:29:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.508571667s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.42847432s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:29:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.428568734s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X.X........X...............................................] duration: 1.590563422s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:29:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.590591856s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.411601141s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:29:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.411635258s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)