Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 2.358424836s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:47:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.358448908s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [...............................XX..XX.......................] duration: 643.835617ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:46:46 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 643.872387ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 2.414209947s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:46:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.414284926s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [................................XX..XX......................] duration: 944.447511ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:46:09 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 944.481664ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.................................XX..XX.....................] duration: 627.23686ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:45:50 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 627.28409ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 2.348720093s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:45:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.348771975s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [..................................XX..XX....................] duration: 720.973765ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:45:09 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 721.0002ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 2.434697235s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:44:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.434725063s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [...................................XX..XX...................] duration: 814.980169ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:44:02 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 815.015952ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 2.392292279s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:43:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.392320446s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [....................................XX..XX..................] duration: 712.239991ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:43:01 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 712.276419ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 2.429713233s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:42:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.429742746s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.....................................XX..XX.................] duration: 708.588235ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:42:02 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 708.646567ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 2.357857112s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:41:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.357883234s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [......................................XX..XX................] duration: 777.760083ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:40:59 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 777.784628ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.......................................XX..XX...............] duration: 663.896575ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:40:37 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 663.986635ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 2.400765289s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:40:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.40087109s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 1.737171601s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:39:50 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.737196979s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [........................................XX..XX..............] duration: 853.566673ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:39:49 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 853.59465ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 2.416572055s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:39:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.416590756s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.........................................XX..XX.............] duration: 610.981061ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:39:29 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 611.010907ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 2.525447799s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:39:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.525478368s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 1.607559685s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:38:43 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.607599438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 2.468759676s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:38:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.468811279s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [..........................................XX..XX............] duration: 636.8348ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:38:32 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 636.85786ms core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 1.620328441s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:38:19 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.620347673s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 2.370611321s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:38:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.370660043s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [...........................................XX..XX...........] duration: 648.100643ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:37:33 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 648.13482ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 2.428011527s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:37:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.428119318s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 1.66392358s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:37:14 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.66394679s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [............................................XX..XX..........] duration: 697.096917ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:36:48 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 697.1259ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 2.541248161s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:36:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.541279221s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.............................................XX..XX.........] duration: 737.288899ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:36:23 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 737.325801ms core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 1.622783209s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:36:19 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.622810179s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.437561957s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:35:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.437590423s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [..............................................XX..XX........] duration: 865.444948ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:35:31 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 865.48585ms core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 2.131566985s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:35:13 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.131631759s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 2.515623614s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:35:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.515656469s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 1.786729928s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:34:39 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.786758575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [...............................................XX..XX.......] duration: 677.536255ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:34:33 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 677.564804ms core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 1.661835231s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:34:05 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.661877032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 2.484493168s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:34:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.484536138s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [................................................XX..XX......] duration: 675.574082ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:33:35 UTC) operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 675.608841ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 2.497383948s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:33:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.497487176s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 1.68852456s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:33:02 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.688558277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.................................................XX..XX.....] duration: 662.562095ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:32:48 UTC) operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 662.589094ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 2.44267666s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:32:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.442708238s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 1.829454256s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:31:48 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.829484718s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [..................................................XX..XX....] duration: 597.910837ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:31:45 UTC) operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 597.938432ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.36913778s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:31:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.369167909s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)