Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.X.....XX......................................] duration: 2.399705698s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:28:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.399728072s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.317438647s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:28:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.317534202s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X.X........X..............................................] duration: 2.021406408s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:28:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.021439806s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.............X.X.....XX.....................................] duration: 2.761203668s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:28:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.761231535s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.349710272s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:27:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.349738025s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X.X........X.............................................] duration: 1.645386678s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:27:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.645419071s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............X.X.....XX....................................] duration: 2.627579578s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:27:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.627611324s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............X.X.....XX...................................] duration: 3.013489318s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:26:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.013600511s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.40138286s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:26:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.401433776s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X.X........X............................................] duration: 2.173671792s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:26:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.173707518s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.328655997s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:26:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.328684149s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [................X.X.....XX..................................] duration: 2.410938977s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:26:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.410967471s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....X.X........X...........................................] duration: 1.738069427s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:26:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.738101727s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X.X........X..........................................] duration: 1.582296104s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:25:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.582374595s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.348995665s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:25:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.349022703s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......X.X........X.........................................] duration: 1.520075463s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:25:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.520126107s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.................X.X.....XX.................................] duration: 2.353564315s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:24:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.353591875s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........X.X........X........................................] duration: 1.68431491s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:24:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.684351898s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.454487715s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:24:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.454518691s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................X.X.....XX................................] duration: 2.543307984s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:23:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.543336413s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........X.X........X.......................................] duration: 1.69306084s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:23:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.693083908s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.471572285s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:23:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.471604164s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........X.X........X......................................] duration: 1.603234658s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:23:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.603290416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...................X.X.....XX...............................] duration: 2.621845579s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:22:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.621874272s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........X.X........X.....................................] duration: 1.953098489s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:22:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.953123838s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............X.X........X....................................] duration: 2.099379312s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:22:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.099405243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.381558149s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:22:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.381583452s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....................X.X.....XX..............................] duration: 2.642403035s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:21:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.64242775s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............X.X........X...................................] duration: 1.600740302s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:21:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.600764095s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............X.X........X..................................] duration: 1.669251075s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:21:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.669307986s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.590410806s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:20:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.590443027s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....................X.X.....XX.............................] duration: 2.513811241s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:20:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.513840754s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............X.X........X.................................] duration: 1.641511841s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:20:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.641563511s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......................X.X.....XX............................] duration: 2.747819248s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:20:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.747842643s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.346104117s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:19:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.346129698s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................X.X........X................................] duration: 1.565057198s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:19:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.565088809s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................X.X........X...............................] duration: 1.580756652s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:19:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.580781994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X.X.....XX...........................] duration: 2.500573605s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:19:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.500599753s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................X.X........X..............................] duration: 1.612533913s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:19:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.612568409s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.376264604s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:18:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.376296685s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X.X........X.............................] duration: 1.573767923s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:18:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.573797747s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X.X.....XX..........................] duration: 2.43338533s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:18:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.433436331s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.337626224s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:18:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.337661529s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................X.X........X............................] duration: 2.060015747s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:17:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.060040585s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.406598355s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:17:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.4066472s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........................X.X.....XX.........................] duration: 2.716880282s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:17:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.716912537s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................X.X........X...........................] duration: 1.615949934s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:17:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.61597359s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.37886816s trace: https://jaeger.dev.storj.io/trace/3037b5621aa45556 FAIL (2025-12-24 00:16:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Dec 23 23:54:28.653481119 (440h26m8.633833756s) [upload=387446]: Starting upload Dec 23 23:54:28.653560538 (440h26m8.633913164s) [upload=387446]: Done starting upload Dec 23 23:54:28.653970600 (440h26m8.634323238s) [upload=387446]: Uploading segments... Dec 23 23:54:28.654542231 (440h26m8.634894867s) [upload=387446]: Got next segment. Inline: false Dec 23 23:54:28.654606061 (440h26m8.634958697s) [upload=387446]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 23:54:28.654610802 (440h26m8.634963437s) [upload=387446]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 23:54:28.764303919 (440h26m8.744656555s) [seg_pos={0 0}, upload=387446]: Begin upload segment... Dec 23 23:54:28.764349788 (440h26m8.744702412s) [seg_pos={0 0}, upload=387446]: Done begin upload segment. Dec 23 23:54:28.764373463 (440h26m8.744726102s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766100234 (440h26m8.746452877s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766520042 (440h26m8.746872763s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.766855018 (440h26m8.747207653s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767029221 (440h26m8.747381857s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767354943 (440h26m8.74770757s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767393434 (440h26m8.747746072s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767786324 (440h26m8.748138967s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.767844064 (440h26m8.748196699s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768207054 (440h26m8.748559693s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768270843 (440h26m8.748623481s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768572879 (440h26m8.748925501s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.768582212 (440h26m8.748934848s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769041319 (440h26m8.74939396s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.769536902 (440h26m8.749889548s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:28.771178487 (440h26m8.751531122s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Uploading piece... Dec 23 23:54:29.382837420 (440h26m9.363190074s) [upload=387446]: Next returned nil segment Dec 23 23:54:29.382855201 (440h26m9.363207846s) [upload=387446]: Waiting for error group managing segments... Dec 23 23:54:29.521893664 (440h26m9.502246302s) [noise=true , address=145.79.227.48:4302 , piece_id=KFJ4NQ5EL6GEVR2IDRYRKIXYZANDFSLDGQEQXQKCNZFQOHDLNBXQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522085811 (440h26m9.502438454s) [noise=true , address=145.79.227.8:4330 , piece_id=2NSOIPDT4BL3ZKESGWOMTWIQB3LI56LVIBYF44Y6XVRNE4EFPP5A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522640139 (440h26m9.502992774s) [noise=true , address=145.79.227.37:4565 , piece_id=JAN2SSF2RNG7567W7Q2J377XVQITEZSVTM5DHJXWEXT6IESLT7QA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.522867197 (440h26m9.50321984s) [noise=true , address=145.79.227.10:4402 , piece_id=NZ6ZERRTUWNY5GIYODL522HXSELKZEIK5NO5US7EOHCRGHRHJXLA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523084080 (440h26m9.503436716s) [noise=true , address=77.90.31.15:4700 , piece_id=ASHOCFCUXCMXKQAOK5KWVEYMOKCNLUTXIYVUB6OSGT4C6IJ6TSOA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.523591357 (440h26m9.503943999s) [noise=true , address=145.79.227.13:4804 , piece_id=HKNINGVX3WLUZJOJAPZX4UUH3LA24QFYPWLUGNSLWLLEM5Y6U3CQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.526738782 (440h26m9.507091418s) [noise=true , address=145.79.227.38:4979 , piece_id=EYAKYVNHWFNF3HSO7EWLUE25ODFK4BQZOG73OV43T4VGTWFIQ74A, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.531580348 (440h26m9.511932995s) [noise=true , address=145.79.227.17:4971 , piece_id=5X3V35DQWGGPOFRZDAUUUGLZT4P5747BMAX2WC4C7YGK7VWOGNMA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.538872130 (440h26m9.519224783s) [noise=true , address=145.79.227.32:4836 , piece_id=TBARG7BYI33TWVNAIXNZNJOWGT2LOV2D6MUNSO2LZ7PF2CEL52AA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.539580702 (440h26m9.519933338s) [noise=true , address=145.79.227.16:4911 , piece_id=U7V3HHWMMFZ5KOOZHROU7N5H3PLFN6VZXA77LAHAX24ITYNZXQTQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:29.550150117 (440h26m9.530502762s) [noise=true , address=145.79.227.41:4086 , piece_id=72VT5HZ7VIMFGFSUXRTASIDEVGTZHCSA2ZURJZGMLCV546COQMFQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:31.443923172 (440h26m11.424275813s) [noise=true , address=145.79.227.34:4905 , piece_id=J4R5GA2PPWLYM5ZF5CGIWV7MRVBCRXFHODO6YQY7XHLGGHWR3YOQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: <nil> Dec 23 23:54:58.379393432 (440h26m38.359746075s) [noise=true , address=45.91.116.14:4192 , piece_id=BOIG25AUVTAA55ORJS74FAB5WL4US24MR4YBKHSUCYMRRUO34JKA, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, address:45.91.116.14:4192): protocol: read tcp 139.59.144.22:55304->45.91.116.14:4192: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379510317 (440h26m38.359862962s) [noise=true , address=45.91.116.5:4300 , piece_id=AZQ7VH3RNG5LJSITP6JQALUMBICDIXWOT3G57K6VILRFP5EKSPKQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, address:45.91.116.5:4300): protocol: rpc: dial tcp 45.91.116.5:4300: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379601437 (440h26m38.359954082s) [noise=true , address=45.91.116.8:4045 , piece_id=AUIGNGNTOBS36HZZEPH2H2GLX6Y6PLXG77I4PYNWE7ZK2ZBN7CSQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 23:54:58.379929794 (440h26m38.36028244s) [noise=true , address=45.91.116.9:4261 , piece_id=TZCKV73YOUNRRAE3UFBAHVSSRIAZJNWK4WUSMCBARWBNR3OXTQJQ, seg_pos={0 0}, upload=387446]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 139.59.144.22:39736->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Dec 23 23:54:58.379969548 (440h26m38.360322171s) [seg_pos={0 0}, upload=387446]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Dec 23 23:54:58.482245207 (440h26m38.462597844s) [upload=387446]: Done uploading segments... Dec 23 23:54:58.482280107 (440h26m38.462632746s) [upload=387446]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.378898972s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........................X.X.....XX........................] duration: 2.461919739s trace: https://jaeger.dev.storj.io/trace/4df74e986e1b71fe FAIL (2025-12-24 00:16:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.461944942s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................X.X........X..........................] duration: 1.650562919s trace: https://jaeger.dev.storj.io/trace/5464670f4ef74d9f FAIL (2025-12-24 00:16:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.650590893s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)