Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 354590 reports (Page 588 of 7092)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.37547005s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:31:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.375503814s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [...................................................XX..XX...] duration: 643.187208ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:31:15 UTC) operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 643.253847ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.317082591s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:30:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.317191598s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [....................................................XX..XX..] duration: 36.49926344s trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:30:48 UTC) operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 36.499291997s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 1.664487585s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:30:37 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.664516526s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 2.364153573s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:30:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.364184293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.337502129s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:29:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.337525279s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................XX..XX.] duration: 729.225419ms trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:29:44 UTC) operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 729.241522ms core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 1.787975707s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:29:44 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.788001792s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XX..XX] duration: 1m0.001414959s trace: https://jaeger.dev.storj.io/trace/2c31a8cce5d1c379 FAIL (2026-03-18 15:29:43 UTC) operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1m0.001427987s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.349494685s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:29:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.349522816s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 1.618061783s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:29:07 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.618089375s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XX..X] duration: 1m0.00154215s trace: https://jaeger.dev.storj.io/trace/7a10a7d1bfad0017 FAIL (2026-03-18 15:28:42 UTC) operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Mar 18 15:27:42.770780343 (191h16m45.786449962s) [upload=183756]: Starting upload Mar 18 15:27:42.770842546 (191h16m45.786512155s) [upload=183756]: Done starting upload Mar 18 15:27:42.770975767 (191h16m45.786645357s) [upload=183756]: Uploading segments... Mar 18 15:27:42.771182474 (191h16m45.786852074s) [upload=183756]: Got next segment. Inline: false Mar 18 15:27:42.771274268 (191h16m45.786943857s) [upload=183756]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:27:42.771300025 (191h16m45.786969629s) [upload=183756]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:28:42.694198777 (191h17m45.709868387s) [upload=183756]: Next segment err: context deadline exceeded Mar 18 15:28:42.694578867 (191h17m45.71024849s) [upload=183756]: Done uploading segments... Mar 18 15:28:42.694630288 (191h17m45.710299891s) [upload=183756]: Upload finished. err: context deadline exceeded Duration: 1m0.001568702s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 1.636735369s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:28:26 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.636762729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.398191546s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:28:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.398221483s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............X.............................................] duration: 1.564307639s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:27:49 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.564334316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.489475138s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:27:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.489512616s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............X............................................] duration: 1.777873511s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:26:45 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.777909932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.504609075s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:26:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.504635712s core uplink us-select-1
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 609.095145ms trace: https://jaeger.dev.storj.io/trace/2a5a9f3e0ba38db1 FAIL (2026-03-18 15:26:28 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 609.124489ms core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 1.776009126s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:26:02 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.776054314s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 992.398534ms trace: https://jaeger.dev.storj.io/trace/2a5a9f3e0ba38db1 FAIL (2026-03-18 15:26:01 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 992.421384ms core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.001364729s trace: https://jaeger.dev.storj.io/trace/2a5a9f3e0ba38db1 FAIL (2026-03-18 15:25:59 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1m0.001381806s core uplink uk-select-1 uk1-placement-17
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 1.624540906s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:25:46 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.62484055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.315944122s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:25:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.316041759s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 1.724094792s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:25:08 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.724121751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12YKhPHptsaNZY8ksCHkfRkP9TBNdeQmjcRdRSZCiZDwoApAShR@uk1.storj.io:7777", Bucket: "orbiter", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.uk1.placement_17) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.000731446s trace: https://jaeger.dev.storj.io/trace/3b39feb3406281a6 FAIL (2026-03-18 15:24:59 UTC) orbiter warning: operation: StatBucket ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== Duration: 1m0.000800886s core uplink uk-select-1 uk1-placement-17
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.436563339s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:24:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.436597012s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 1.713792133s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:24:24 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.713843449s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.350691226s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:24:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.350727865s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.579741154s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:23:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.57976908s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 1.89787754s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:23:41 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.898273189s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 1.744547214s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:22:55 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.744577474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.357386543s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:22:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.35741642s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 1.727715041s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:21:41 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.727740047s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.861061648s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:21:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 2.861107216s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.0026338s trace: https://jaeger.dev.storj.io/trace/21b23d7305e037a FAIL (2026-03-18 15:21:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, Piece ID: 3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA) context deadline exceeded; piecestore: (Node ID: 12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, Piece ID: ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA) context deadline exceeded; piecestore: (Node ID: 128kFn1ZsqrdV936j55GuKpnHdERmtBxcL4B9bZrnjeXMaLpfsP, Piece ID: YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ) context deadline exceeded ======LOGS====== upload=183651]: Starting upload Mar 18 15:20:54.303457115 (191h9m57.319126727s) [upload=183651]: Done starting upload Mar 18 15:20:54.303665912 (191h9m57.319335513s) [upload=183651]: Uploading segments... Mar 18 15:20:54.303965721 (191h9m57.319635311s) [upload=183651]: Got next segment. Inline: false Mar 18 15:20:54.304013634 (191h9m57.319683246s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectBegin Mar 18 15:20:54.304021720 (191h9m57.319691334s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentBegin Mar 18 15:20:54.662509110 (191h9m57.67817871s) [seg_pos={0 0}, upload=183651]: Begin upload segment... Mar 18 15:20:54.662562678 (191h9m57.678232268s) [seg_pos={0 0}, upload=183651]: Done begin upload segment. Mar 18 15:20:54.662592279 (191h9m57.67826188s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664091904 (191h9m57.679761511s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.664760971 (191h9m57.680430574s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.665304567 (191h9m57.68097417s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666050630 (191h9m57.68172024s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.666631225 (191h9m57.682300831s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667041838 (191h9m57.68271144s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.667546703 (191h9m57.683216304s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668012057 (191h9m57.683681657s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668448937 (191h9m57.684118543s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.668687476 (191h9m57.684357085s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.669348163 (191h9m57.685017763s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670185533 (191h9m57.685855138s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670284943 (191h9m57.685954546s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.670802120 (191h9m57.686471739s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:54.671365214 (191h9m57.687034823s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Uploading piece... Mar 18 15:20:55.553326060 (191h9m58.568995658s) [upload=183651]: Next returned nil segment Mar 18 15:20:55.553350353 (191h9m58.569019965s) [upload=183651]: Waiting for error group managing segments... Mar 18 15:20:55.731225208 (191h9m58.746894812s) [noise=true , address=79.127.219.45:4855 , piece_id=EYM67437Z7OQ3J4VRCVH53FCRBRAEBERDK4Y5MPBM7EGPFHAIXUQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.877239140 (191h9m58.89290874s) [noise=true , address=145.79.227.7:4297 , piece_id=NLAHTGYCTU6KA4KEYXYNXJGIJCHPQH5LVD3A5LMT3G4S2X6HI6WQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.881970180 (191h9m58.897639791s) [noise=true , address=145.79.227.19:4083 , piece_id=GNMET2EJCABKI3ZEEY3U5SABD5PR4KAUCXVWEUJYRN7TO33QHEAA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.890259873 (191h9m58.905929478s) [noise=true , address=45.91.116.7:4012 , piece_id=KYPUR2EBM2QJWXJQG3VCHCCZLWCGLEF4HDXEJRU4YEXCHDWSTEDA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.895733842 (191h9m58.911403443s) [noise=true , address=145.79.227.6:4262 , piece_id=3J4NH2MLBKVMF7WBUQCQGUJPSONQGTXEUHIIZBFMN5P6I2N4QXDQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.922327749 (191h9m58.937997359s) [noise=true , address=145.79.227.14:4837 , piece_id=PQ2STQPAJZQTANRAMDKBTPFNSWGWV2Z4PF6MGUESOV7V35XI2Z6Q, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:55.935924481 (191h9m58.951594082s) [noise=true , address=145.79.227.50:4373 , piece_id=ZYW7ES3I3FN5TSJNJ57WX4JZZ7WCXBEXDQEEZGHXLSAMVV57JYUA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.490019568 (191h9m59.505689171s) [noise=true , address=77.90.31.14:4661 , piece_id=RIYVADNOLZAQSI55TC6QNMDHQRCCX6VJGWCXVHGEPNJWTYOLX5RQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.498992750 (191h9m59.51466235s) [noise=true , address=145.79.227.42:4123 , piece_id=C3AUR2K7GZPBX2O7HMPF3NS7SKPRJSCFHIFMARF5MNML32QPCJLA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.503673826 (191h9m59.519343429s) [noise=true , address=145.79.227.35:4484 , piece_id=4B5H6IWVZKFJFNPT2YFHHUJ7XPUK6JT7DHDSVYZNRXBGZUTM72TQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.584581917 (191h9m59.600251521s) [noise=true , address=45.91.116.13:4154 , piece_id=5C7TL3GT7YEUIUF6MQLHS5YQPWGU4HMVXJ3RXPHHQ4EUQ7WN2NBA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.596338514 (191h9m59.612008115s) [noise=true , address=145.79.227.49:4341 , piece_id=RBOQ42IOZAAZXE35PXOW6FITIIPUED6HQOKX3MZ3MJQJN7Z34NJQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719952247 (191h9m59.735621857s) [noise=true , address=145.79.227.29:4945 , piece_id=YAW47YHT74Q5PRWVOGBDEIXGGCXWJBQZK4PKC334IRGB5WBY44OQ, seg_pos={0 0}, upload=183651]: Done uploading piece. err: <nil> Mar 18 15:20:56.719982026 (191h9m59.735651634s) [seg_pos={0 0}, upload=183651]: Satisfied cohort requirements after 13 pieces Mar 18 15:20:56.720279864 (191h9m59.735949473s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:42930->45.91.116.11:4120: use of closed network connection Mar 18 15:20:56.720290575 (191h9m59.735960184s) [noise=true , address=45.91.116.11:4120 , piece_id=GR4SEE5RUEY6BZRC3NM22LRTB5LRD7XEQARZ3HNV73BWEHW77Q6A, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721072812 (191h9m59.736742422s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127hafVDbun7FoG4JdcogBmRNT6rhL2QQbLDLstFnx7HCWyiC9N): protocol: expected piece hash; context canceled; EOF Mar 18 15:20:56.721085662 (191h9m59.736755272s) [noise=true , address=145.79.227.30:4764 , piece_id=OBRMP5YGOPFDUHOMR5FKU2GVWAF7EBC7QSNIXDZWFOM4TOWVXDOA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721315732 (191h9m59.736985341s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F): protocol: read tcp 139.59.144.22:41040->145.79.227.41:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Mar 18 15:20:56.721327176 (191h9m59.736996776s) [noise=true , address=145.79.227.41:4090 , piece_id=6JKG7VBPB54GTHPNX6W37KD64QK6SJAYL6UQSW5U3RAWL52SHFGA, seg_pos={0 0}, upload=183651]: Long tail cancellation due to optimal threshold Mar 18 15:20:56.721354703 (191h9m59.737024289s) [seg_pos={0 0}, upload=183651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Mar 18 15:20:56.721444047 (191h9m59.737113648s) [upload=183651]: Flush batch item: pb.BatchRequestItem_SegmentCommit Mar 18 15:20:56.721450536 (191h9m59.737120136s) [upload=183651]: Flush batch item: pb.BatchRequestItem_ObjectCommit Mar 18 15:20:56.945499978 (191h9m59.961169586s) [upload=183651]: Done uploading segments... Mar 18 15:20:56.945534492 (191h9m59.961204094s) [upload=183651]: Upload finished. err: <nil> Duration: 30.002656712s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.4859934s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:20:58 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 3.486038019s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 1.661068074s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:20:41 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.661095662s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 1.778474007s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:19:59 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.779194238s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 2.273943566s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:19:23 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.273973919s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 1.750321548s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:18:42 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.750356082s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 1.712902157s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:18:23 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.712924277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 1.76325187s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:18:04 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.763343507s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 1.770899087s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:16:54 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.77092571s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 2.170075583s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:16:26 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.170104097s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 1.864753559s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:16:01 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.864862734s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 1.664753087s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:15:10 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.664780697s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 2.338935784s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:14:48 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 2.339324572s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 1.758042377s trace: https://jaeger.dev.storj.io/trace/34c90c99cd86d8a8 FAIL (2026-03-18 15:14:12 UTC) orbiter warning: Post "https://auth.storjshare.io/v1/access": context deadline exceeded Duration: 1.758068236s core uplink us-select-1 us1-placement-20

1
...
586
587
588
589
590
...
7092