Orbiter History

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [..........XX..XX............................................] duration: 703.050641ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 16:02:19 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 703.082593ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.400721637s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 16:01:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.400750369s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.430240342s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 16:01:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.43026903s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [...........XX..XX...........................................] duration: 707.664678ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 16:01:17 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 707.697498ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.513193263s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 16:01:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.513224968s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [............XX..XX..........................................] duration: 779.553459ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 16:01:02 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 779.641174ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.............XX..XX.........................................] duration: 750.56727ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 16:00:37 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 750.58563ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.322692286s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:59:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.322799075s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [..............XX..XX........................................] duration: 712.707017ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:59:29 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 712.74585ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.366381655s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:58:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.366425922s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [...............XX..XX.......................................] duration: 660.396986ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:58:26 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 660.437202ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.352712482s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:58:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.352740497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.367885805s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:57:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.367930089s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [................XX..XX......................................] duration: 671.138079ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:57:37 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 671.168421ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.327207058s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:57:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.327241861s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.................XX..XX.....................................] duration: 699.933331ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:57:19 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 699.961846ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.406991922s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:56:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.407010484s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [..................XX..XX....................................] duration: 659.05092ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:56:45 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 659.136032ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [...................XX..XX...................................] duration: 750.179391ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:56:28 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 750.291515ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.360842914s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:56:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.360874563s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [....................XX..XX..................................] duration: 811.606894ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:55:51 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 811.639645ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.310419381s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:55:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.310444384s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.....................XX..XX.................................] duration: 673.265857ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:54:53 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 673.30182ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.499354179s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:54:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.499383589s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [......................XX..XX................................] duration: 659.47926ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:54:00 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 659.518605ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.......................XX..XX...............................] duration: 674.84732ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:53:20 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 674.875778ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.267009887s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:53:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.267037799s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [........................XX..XX..............................] duration: 650.908971ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:53:02 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 650.973525ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 2.382756348s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:52:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.382795087s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.........................XX..XX.............................] duration: 581.024693ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:51:53 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 581.09783ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 2.36435798s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:51:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.364389635s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [..........................XX..XX............................] duration: 661.633076ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:51:03 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 661.743387ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 2.358805208s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:50:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.358853561s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [...........................XX..XX...........................] duration: 598.988957ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:50:28 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 599.02045ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 2.35293997s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:49:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.35297278s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 2.400183021s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:49:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.400227073s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [............................XX..XX..........................] duration: 699.606123ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:49:19 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 699.644145ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.............................XX..XX.........................] duration: 696.750009ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:48:39 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 696.78728ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 2.362474333s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:48:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.362537201s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [..............................XX..XX........................] duration: 719.634118ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:48:01 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 719.664022ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.358424836s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:47:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.358448908s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [...............................XX..XX.......................] duration: 643.835617ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:46:46 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 643.872387ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.414209947s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:46:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.414284926s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [................................XX..XX......................] duration: 944.447511ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:46:09 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 944.481664ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.................................XX..XX.....................] duration: 627.23686ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:45:50 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 627.28409ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.348720093s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:45:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.348771975s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [..................................XX..XX....................] duration: 720.973765ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:45:09 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 721.0002ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.434697235s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:44:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.434725063s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [...................................XX..XX...................] duration: 814.980169ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:44:02 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 815.015952ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.392292279s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:43:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.392320446s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)