Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........................X.X.....XX.......................] duration: 2.471007277s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:16:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.471107789s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.291853944s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:16:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.291904541s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................X.X........X.........................] duration: 1.580816197s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:15:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.580844596s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............................X.X.....XX......................] duration: 2.651410419s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:15:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.651515532s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.405431334s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:15:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.405470459s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............................X.X.....XX.....................] duration: 2.637508051s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:15:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.637543884s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................X.X........X........................] duration: 2.089657621s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:15:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.089683326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.397629578s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:14:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.397718955s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................X.X........X.......................] duration: 1.528899248s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:14:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.528924515s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................X.X........X......................] duration: 1.655155395s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:14:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.655181761s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................X.X.....XX....................] duration: 2.857574277s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:14:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.857667089s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................X.X........X.....................] duration: 2.065582431s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:13:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.065600036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.377205407s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:13:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.377237729s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................X.X.....XX...................] duration: 2.650000486s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:13:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.650029038s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.33879159s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:13:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.338825944s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................X.X........X....................] duration: 1.588026559s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:13:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.588054386s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................X.X........X...................] duration: 1.527295822s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:13:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.527323941s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................X.X.....XX..................] duration: 2.551726176s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:12:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.551828588s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................X.X........X..................] duration: 1.595260992s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:12:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.595347568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.397888488s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:12:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.397920242s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................X.X........X.................] duration: 1.6017063s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:12:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.60183387s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................................X.X.....XX.................] duration: 2.517491797s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:11:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.517546601s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.407903456s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:11:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.408000744s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................................X.X........X................] duration: 2.497257648s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:11:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.49728838s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.327718315s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:11:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.327743323s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................X.X.....XX................] duration: 2.774061764s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:10:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.774096494s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.352536075s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:10:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.352618227s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................................X.X........X...............] duration: 1.679593883s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:10:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.679628126s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.340168289s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:09:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.340197299s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................................X.X.....XX...............] duration: 2.500665789s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:09:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.500685385s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................................X.X........X..............] duration: 1.565070993s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:09:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.565091359s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................................X.X.....XX..............] duration: 2.527217087s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:09:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.527241908s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.381280836s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:09:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.381306918s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................................X.X........X.............] duration: 1.843758012s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:08:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.843787452s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................................X.X.....XX.............] duration: 2.609160178s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:08:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.609189671s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................................X.X........X............] duration: 1.664299042s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:08:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.6643489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.505185852s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:08:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.505203991s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................................X.X.....XX............] duration: 2.746834329s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:08:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.746864493s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................................X.X........X...........] duration: 1.633014752s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:07:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.633044306s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................................X.X.....XX...........] duration: 2.534154497s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:07:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.534178687s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................................X.X........X..........] duration: 2.018610855s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:07:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.018682973s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.386857675s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:07:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.386882832s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................................X.X.....XX..........] duration: 2.767743193s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:06:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.767806483s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................................X.X.....XX.........] duration: 2.527017087s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:06:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.527045538s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.320698293s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:06:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.320731536s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................................X.X........X.........] duration: 1.624828082s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:06:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.624854466s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................................X.X.....XX........] duration: 2.46219472s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:06:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.462243919s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................................X.X........X........] duration: 1.5634666s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:05:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.563517825s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.529949541s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:05:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.529979422s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................................X.X........X.......] duration: 1.572380284s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:05:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.572406103s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)