Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................................X.X.....XX.......] duration: 2.767189424s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:05:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.767243031s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................................X.X.....XX......] duration: 2.505481157s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:04:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.505515015s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.388515231s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:04:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.388597881s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................................X.X........X......] duration: 1.597364411s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:04:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.597390488s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................................X.X.....XX.....] duration: 2.549567417s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:04:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.549599509s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.456371544s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:03:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.456408461s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................X.X.....XX....] duration: 2.71647602s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:03:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.716647199s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.337836205s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:03:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.337869964s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................................X.X........X.....] duration: 1.583385538s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:03:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.583450331s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................................X.X........X....] duration: 1.54036668s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:02:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.540399758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.772357979s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:02:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.772404058s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................X.X.....XX...] duration: 2.571420848s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:02:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.57144874s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................................X.X........X...] duration: 1.674902518s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:01:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.674939165s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.417333832s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:01:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.417365485s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................X.X.....XX..] duration: 2.712777973s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:01:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.712808559s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................................X.X........X..] duration: 1.663688731s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:01:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.663715301s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................................X.X.....XX.] duration: 2.555505047s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:01:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.555539578s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................................X.X........X.] duration: 1.722819188s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:00:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.722860466s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 4.103686465s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:00:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 4.103718942s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................X.X.....XX] duration: 30.003816848s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:00:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.0038423s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 3.54619925s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:00:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 3.546241622s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X.X.....X] duration: 30.002199676s trace: https://jaeger.dev.storj.io/trace/49942e7941ae32ba FAIL (2025-12-24 00:00:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002226178s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................................................X.X........X] duration: 5.001085809s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-23 23:59:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001112538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................................................X.X........] duration: 2.00473624s trace: https://jaeger.dev.storj.io/trace/77804f5fa33cf456 FAIL (2025-12-23 23:59:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.004832104s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.373397735s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-23 23:58:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.373424611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.376684705s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-23 23:58:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.376711966s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................................................X.X.......] duration: 1.673676522s trace: https://jaeger.dev.storj.io/trace/77804f5fa33cf456 FAIL (2025-12-23 23:58:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.673702163s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 2.42851275s trace: https://jaeger.dev.storj.io/trace/2e5b1c0889b6b7cd FAIL (2025-12-23 23:58:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.428537325s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.393700256s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-23 23:57:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.393726838s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................................................X.X......] duration: 1.51334415s trace: https://jaeger.dev.storj.io/trace/77804f5fa33cf456 FAIL (2025-12-23 23:57:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.513364968s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 2.51506663s trace: https://jaeger.dev.storj.io/trace/2e5b1c0889b6b7cd FAIL (2025-12-23 23:57:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.515174138s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.347354747s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-23 23:57:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.347381521s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................................................X.X.....] duration: 1.634982549s trace: https://jaeger.dev.storj.io/trace/77804f5fa33cf456 FAIL (2025-12-23 23:57:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.63500485s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 2.566270941s trace: https://jaeger.dev.storj.io/trace/2e5b1c0889b6b7cd FAIL (2025-12-23 23:56:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.56629868s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................................................X.X....] duration: 1.64932995s trace: https://jaeger.dev.storj.io/trace/77804f5fa33cf456 FAIL (2025-12-23 23:56:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.649357829s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.374670769s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-23 23:56:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.374700088s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................................................X.X...] duration: 1.540947815s trace: https://jaeger.dev.storj.io/trace/77804f5fa33cf456 FAIL (2025-12-23 23:56:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.540973528s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 2.645280302s trace: https://jaeger.dev.storj.io/trace/2e5b1c0889b6b7cd FAIL (2025-12-23 23:56:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.645310415s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.382768885s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-23 23:56:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.382795375s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................................................X.X..] duration: 1.649249508s trace: https://jaeger.dev.storj.io/trace/77804f5fa33cf456 FAIL (2025-12-23 23:55:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.649341373s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................................................X.X.] duration: 2.062235255s trace: https://jaeger.dev.storj.io/trace/77804f5fa33cf456 FAIL (2025-12-23 23:55:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.062369418s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.520617733s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-23 23:55:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.520649371s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 17.697060296s trace: https://jaeger.dev.storj.io/trace/2e5b1c0889b6b7cd FAIL (2025-12-23 23:55:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.697152417s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.104139368s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-23 23:54:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.104163401s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 30.001786731s trace: https://jaeger.dev.storj.io/trace/2e5b1c0889b6b7cd FAIL (2025-12-23 23:54:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.0018228s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................................................X.X] duration: 5.001512159s trace: https://jaeger.dev.storj.io/trace/77804f5fa33cf456 FAIL (2025-12-23 23:54:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001538305s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.514956819s trace: https://jaeger.dev.storj.io/trace/146f21cf8820c4b1 FAIL (2025-12-23 23:53:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.51498047s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X.........................................................X.] duration: 1.731812006s trace: https://jaeger.dev.storj.io/trace/58a811fb876c5e03 FAIL (2025-12-23 23:53:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.731840059s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001354791s trace: https://jaeger.dev.storj.io/trace/146f21cf8820c4b1 FAIL (2025-12-23 23:53:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001380168s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X.........................................................X] duration: 5.001540993s trace: https://jaeger.dev.storj.io/trace/58a811fb876c5e03 FAIL (2025-12-23 23:52:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001563823s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)