Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....................................X..X...................] duration:
15.177179211s
FAIL (2025-07-09 00:54:14 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
entCommit
Jul 9 00:38:54.141867066 (375h27m54.025280684s) [upload=300248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:54.286588243 (375h27m54.170001881s) [upload=300248]: Done uploading segments...
Jul 9 00:38:54.286633475 (375h27m54.170047105s) [upload=300248]: Upload finished. err: <nil>
Jul 9 00:39:49.853458099 (375h28m49.736871729s) [upload=300262]: Starting upload
Jul 9 00:39:49.853564556 (375h28m49.736978174s) [upload=300262]: Done starting upload
Jul 9 00:39:49.853754759 (375h28m49.737168394s) [upload=300262]: Uploading segments...
Jul 9 00:39:49.854072231 (375h28m49.737485857s) [upload=300262]: Got next segment. Inline: false
Jul 9 00:39:49.854139888 (375h28m49.737553515s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:39:49.854145670 (375h28m49.737559297s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:39:49.964717413 (375h28m49.848131039s) [seg_pos={0 0}, upload=300262]: Begin upload segment...
Jul 9 00:39:49.964772224 (375h28m49.848185839s) [seg_pos={0 0}, upload=300262]: Done begin upload segment.
Jul 9 00:39:49.964909767 (375h28m49.848323393s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965422318 (375h28m49.848835947s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965585955 (375h28m49.848999581s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966217091 (375h28m49.849630727s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966292916 (375h28m49.849706542s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967093655 (375h28m49.85050729s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967169721 (375h28m49.850583348s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967940005 (375h28m49.851353634s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.968588971 (375h28m49.8520026s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969080450 (375h28m49.852494085s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969563775 (375h28m49.852977409s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.970736354 (375h28m49.854149991s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.971229734 (375h28m49.85464336s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.972867679 (375h28m49.856281317s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.978267895 (375h28m49.861681529s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.986076032 (375h28m49.869489662s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:50.595435555 (375h28m50.478849173s) [upload=300262]: Next returned nil segment
Jul 9 00:39:50.595455642 (375h28m50.478869272s) [upload=300262]: Waiting for error group managing segments...
Jul 9 00:39:50.730628340 (375h28m50.614041956s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.731179161 (375h28m50.614592788s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.732951876 (375h28m50.616365493s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.738403976 (375h28m50.621817592s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739011956 (375h28m50.622425573s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739658988 (375h28m50.623072606s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.766298764 (375h28m50.649712392s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.754942777 (375h28m51.638356391s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.990915358 (375h28m51.87432899s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:13.196278807 (375h29m13.079692437s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:19.575930850 (375h29m19.459344476s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576005813 (375h29m19.459419439s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576051734 (375h29m19.459465361s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576066765 (375h29m19.459480398s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576255242 (375h29m19.459668869s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection
Jul 9 00:40:19.576293041 (375h29m19.459706667s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection
Jul 9 00:40:19.576316511 (375h29m19.45973014s) [seg_pos={0 0}, upload=300262]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 9 00:40:19.685159343 (375h29m19.568572971s) [upload=300262]: Done uploading segments...
Jul 9 00:40:19.685187481 (375h29m19.568601108s) [upload=300262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 15.177209377s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [....XX...X..............X....X..X......X.............XX.X.X.] duration:
25.675961469s
FAIL (2025-07-09 00:54:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 25.67598713s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
12 out of 60 is failed (threshold=3) [.X.................X...........XX.X..X..X..XX.X...........XX] duration:
1m0.001552808s
FAIL (2025-07-09 00:53:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001568283s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.....XX...X..............X....X..X......X.............XX.X.X] duration:
30.000365871s
FAIL (2025-07-09 00:53:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000395808s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
20.186806521s
FAIL (2025-07-09 00:53:21 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 20.186930442s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................X..X..................] duration:
2.298698425s
FAIL (2025-07-09 00:53:09 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
entCommit
Jul 9 00:38:54.141867066 (375h27m54.025280684s) [upload=300248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:54.286588243 (375h27m54.170001881s) [upload=300248]: Done uploading segments...
Jul 9 00:38:54.286633475 (375h27m54.170047105s) [upload=300248]: Upload finished. err: <nil>
Jul 9 00:39:49.853458099 (375h28m49.736871729s) [upload=300262]: Starting upload
Jul 9 00:39:49.853564556 (375h28m49.736978174s) [upload=300262]: Done starting upload
Jul 9 00:39:49.853754759 (375h28m49.737168394s) [upload=300262]: Uploading segments...
Jul 9 00:39:49.854072231 (375h28m49.737485857s) [upload=300262]: Got next segment. Inline: false
Jul 9 00:39:49.854139888 (375h28m49.737553515s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:39:49.854145670 (375h28m49.737559297s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:39:49.964717413 (375h28m49.848131039s) [seg_pos={0 0}, upload=300262]: Begin upload segment...
Jul 9 00:39:49.964772224 (375h28m49.848185839s) [seg_pos={0 0}, upload=300262]: Done begin upload segment.
Jul 9 00:39:49.964909767 (375h28m49.848323393s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965422318 (375h28m49.848835947s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965585955 (375h28m49.848999581s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966217091 (375h28m49.849630727s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966292916 (375h28m49.849706542s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967093655 (375h28m49.85050729s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967169721 (375h28m49.850583348s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967940005 (375h28m49.851353634s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.968588971 (375h28m49.8520026s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969080450 (375h28m49.852494085s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969563775 (375h28m49.852977409s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.970736354 (375h28m49.854149991s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.971229734 (375h28m49.85464336s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.972867679 (375h28m49.856281317s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.978267895 (375h28m49.861681529s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.986076032 (375h28m49.869489662s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:50.595435555 (375h28m50.478849173s) [upload=300262]: Next returned nil segment
Jul 9 00:39:50.595455642 (375h28m50.478869272s) [upload=300262]: Waiting for error group managing segments...
Jul 9 00:39:50.730628340 (375h28m50.614041956s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.731179161 (375h28m50.614592788s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.732951876 (375h28m50.616365493s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.738403976 (375h28m50.621817592s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739011956 (375h28m50.622425573s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739658988 (375h28m50.623072606s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.766298764 (375h28m50.649712392s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.754942777 (375h28m51.638356391s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.990915358 (375h28m51.87432899s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:13.196278807 (375h29m13.079692437s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:19.575930850 (375h29m19.459344476s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576005813 (375h29m19.459419439s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576051734 (375h29m19.459465361s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576066765 (375h29m19.459480398s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576255242 (375h29m19.459668869s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection
Jul 9 00:40:19.576293041 (375h29m19.459706667s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection
Jul 9 00:40:19.576316511 (375h29m19.45973014s) [seg_pos={0 0}, upload=300262]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 9 00:40:19.685159343 (375h29m19.568572971s) [upload=300262]: Done uploading segments...
Jul 9 00:40:19.685187481 (375h29m19.568601108s) [upload=300262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.298721786s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................X..X.................] duration:
29.824025063s
FAIL (2025-07-09 00:53:07 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
entCommit
Jul 9 00:38:54.141867066 (375h27m54.025280684s) [upload=300248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:54.286588243 (375h27m54.170001881s) [upload=300248]: Done uploading segments...
Jul 9 00:38:54.286633475 (375h27m54.170047105s) [upload=300248]: Upload finished. err: <nil>
Jul 9 00:39:49.853458099 (375h28m49.736871729s) [upload=300262]: Starting upload
Jul 9 00:39:49.853564556 (375h28m49.736978174s) [upload=300262]: Done starting upload
Jul 9 00:39:49.853754759 (375h28m49.737168394s) [upload=300262]: Uploading segments...
Jul 9 00:39:49.854072231 (375h28m49.737485857s) [upload=300262]: Got next segment. Inline: false
Jul 9 00:39:49.854139888 (375h28m49.737553515s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:39:49.854145670 (375h28m49.737559297s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:39:49.964717413 (375h28m49.848131039s) [seg_pos={0 0}, upload=300262]: Begin upload segment...
Jul 9 00:39:49.964772224 (375h28m49.848185839s) [seg_pos={0 0}, upload=300262]: Done begin upload segment.
Jul 9 00:39:49.964909767 (375h28m49.848323393s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965422318 (375h28m49.848835947s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965585955 (375h28m49.848999581s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966217091 (375h28m49.849630727s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966292916 (375h28m49.849706542s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967093655 (375h28m49.85050729s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967169721 (375h28m49.850583348s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967940005 (375h28m49.851353634s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.968588971 (375h28m49.8520026s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969080450 (375h28m49.852494085s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969563775 (375h28m49.852977409s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.970736354 (375h28m49.854149991s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.971229734 (375h28m49.85464336s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.972867679 (375h28m49.856281317s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.978267895 (375h28m49.861681529s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.986076032 (375h28m49.869489662s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:50.595435555 (375h28m50.478849173s) [upload=300262]: Next returned nil segment
Jul 9 00:39:50.595455642 (375h28m50.478869272s) [upload=300262]: Waiting for error group managing segments...
Jul 9 00:39:50.730628340 (375h28m50.614041956s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.731179161 (375h28m50.614592788s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.732951876 (375h28m50.616365493s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.738403976 (375h28m50.621817592s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739011956 (375h28m50.622425573s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739658988 (375h28m50.623072606s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.766298764 (375h28m50.649712392s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.754942777 (375h28m51.638356391s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.990915358 (375h28m51.87432899s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:13.196278807 (375h29m13.079692437s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:19.575930850 (375h29m19.459344476s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576005813 (375h29m19.459419439s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576051734 (375h29m19.459465361s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576066765 (375h29m19.459480398s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576255242 (375h29m19.459668869s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection
Jul 9 00:40:19.576293041 (375h29m19.459706667s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection
Jul 9 00:40:19.576316511 (375h29m19.45973014s) [seg_pos={0 0}, upload=300262]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 9 00:40:19.685159343 (375h29m19.568572971s) [upload=300262]: Done uploading segments...
Jul 9 00:40:19.685187481 (375h29m19.568601108s) [upload=300262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 29.824055699s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
12 out of 60 is failed (threshold=3) [X.X.................X...........XX.X..X..X..XX.X...........X] duration:
1m0.00102605s
FAIL (2025-07-09 00:52:32 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001056325s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
17.407637571s
FAIL (2025-07-09 00:52:31 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 17.407658693s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [......XX...X..............X....X..X......X.............XX.X.] duration:
20.905480544s
FAIL (2025-07-09 00:52:15 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.905504686s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
26.805495231s
FAIL (2025-07-09 00:52:13 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 26.805546271s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.......XX...X..............X....X..X......X.............XX.X] duration:
30.001500175s
FAIL (2025-07-09 00:51:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001526007s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................X..X................] duration:
2.497920244s
FAIL (2025-07-09 00:51:26 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
entCommit
Jul 9 00:38:54.141867066 (375h27m54.025280684s) [upload=300248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:54.286588243 (375h27m54.170001881s) [upload=300248]: Done uploading segments...
Jul 9 00:38:54.286633475 (375h27m54.170047105s) [upload=300248]: Upload finished. err: <nil>
Jul 9 00:39:49.853458099 (375h28m49.736871729s) [upload=300262]: Starting upload
Jul 9 00:39:49.853564556 (375h28m49.736978174s) [upload=300262]: Done starting upload
Jul 9 00:39:49.853754759 (375h28m49.737168394s) [upload=300262]: Uploading segments...
Jul 9 00:39:49.854072231 (375h28m49.737485857s) [upload=300262]: Got next segment. Inline: false
Jul 9 00:39:49.854139888 (375h28m49.737553515s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:39:49.854145670 (375h28m49.737559297s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:39:49.964717413 (375h28m49.848131039s) [seg_pos={0 0}, upload=300262]: Begin upload segment...
Jul 9 00:39:49.964772224 (375h28m49.848185839s) [seg_pos={0 0}, upload=300262]: Done begin upload segment.
Jul 9 00:39:49.964909767 (375h28m49.848323393s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965422318 (375h28m49.848835947s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965585955 (375h28m49.848999581s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966217091 (375h28m49.849630727s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966292916 (375h28m49.849706542s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967093655 (375h28m49.85050729s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967169721 (375h28m49.850583348s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967940005 (375h28m49.851353634s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.968588971 (375h28m49.8520026s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969080450 (375h28m49.852494085s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969563775 (375h28m49.852977409s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.970736354 (375h28m49.854149991s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.971229734 (375h28m49.85464336s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.972867679 (375h28m49.856281317s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.978267895 (375h28m49.861681529s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.986076032 (375h28m49.869489662s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:50.595435555 (375h28m50.478849173s) [upload=300262]: Next returned nil segment
Jul 9 00:39:50.595455642 (375h28m50.478869272s) [upload=300262]: Waiting for error group managing segments...
Jul 9 00:39:50.730628340 (375h28m50.614041956s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.731179161 (375h28m50.614592788s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.732951876 (375h28m50.616365493s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.738403976 (375h28m50.621817592s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739011956 (375h28m50.622425573s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739658988 (375h28m50.623072606s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.766298764 (375h28m50.649712392s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.754942777 (375h28m51.638356391s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.990915358 (375h28m51.87432899s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:13.196278807 (375h29m13.079692437s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:19.575930850 (375h29m19.459344476s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576005813 (375h29m19.459419439s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576051734 (375h29m19.459465361s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576066765 (375h29m19.459480398s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576255242 (375h29m19.459668869s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection
Jul 9 00:40:19.576293041 (375h29m19.459706667s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection
Jul 9 00:40:19.576316511 (375h29m19.45973014s) [seg_pos={0 0}, upload=300262]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 9 00:40:19.685159343 (375h29m19.568572971s) [upload=300262]: Done uploading segments...
Jul 9 00:40:19.685187481 (375h29m19.568601108s) [upload=300262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.497963098s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
20.771199294s
FAIL (2025-07-09 00:51:08 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 20.771231617s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
12 out of 60 is failed (threshold=3) [XX.X.................X...........XX.X..X..X..XX.X...........] duration:
33.209449056s
FAIL (2025-07-09 00:50:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 33.209470805s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................X..X...............] duration:
2.40185602s
FAIL (2025-07-09 00:50:54 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
entCommit
Jul 9 00:38:54.141867066 (375h27m54.025280684s) [upload=300248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:54.286588243 (375h27m54.170001881s) [upload=300248]: Done uploading segments...
Jul 9 00:38:54.286633475 (375h27m54.170047105s) [upload=300248]: Upload finished. err: <nil>
Jul 9 00:39:49.853458099 (375h28m49.736871729s) [upload=300262]: Starting upload
Jul 9 00:39:49.853564556 (375h28m49.736978174s) [upload=300262]: Done starting upload
Jul 9 00:39:49.853754759 (375h28m49.737168394s) [upload=300262]: Uploading segments...
Jul 9 00:39:49.854072231 (375h28m49.737485857s) [upload=300262]: Got next segment. Inline: false
Jul 9 00:39:49.854139888 (375h28m49.737553515s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:39:49.854145670 (375h28m49.737559297s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:39:49.964717413 (375h28m49.848131039s) [seg_pos={0 0}, upload=300262]: Begin upload segment...
Jul 9 00:39:49.964772224 (375h28m49.848185839s) [seg_pos={0 0}, upload=300262]: Done begin upload segment.
Jul 9 00:39:49.964909767 (375h28m49.848323393s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965422318 (375h28m49.848835947s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965585955 (375h28m49.848999581s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966217091 (375h28m49.849630727s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966292916 (375h28m49.849706542s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967093655 (375h28m49.85050729s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967169721 (375h28m49.850583348s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967940005 (375h28m49.851353634s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.968588971 (375h28m49.8520026s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969080450 (375h28m49.852494085s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969563775 (375h28m49.852977409s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.970736354 (375h28m49.854149991s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.971229734 (375h28m49.85464336s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.972867679 (375h28m49.856281317s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.978267895 (375h28m49.861681529s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.986076032 (375h28m49.869489662s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:50.595435555 (375h28m50.478849173s) [upload=300262]: Next returned nil segment
Jul 9 00:39:50.595455642 (375h28m50.478869272s) [upload=300262]: Waiting for error group managing segments...
Jul 9 00:39:50.730628340 (375h28m50.614041956s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.731179161 (375h28m50.614592788s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.732951876 (375h28m50.616365493s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.738403976 (375h28m50.621817592s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739011956 (375h28m50.622425573s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739658988 (375h28m50.623072606s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.766298764 (375h28m50.649712392s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.754942777 (375h28m51.638356391s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.990915358 (375h28m51.87432899s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:13.196278807 (375h29m13.079692437s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:19.575930850 (375h29m19.459344476s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576005813 (375h29m19.459419439s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576051734 (375h29m19.459465361s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576066765 (375h29m19.459480398s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576255242 (375h29m19.459668869s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection
Jul 9 00:40:19.576293041 (375h29m19.459706667s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection
Jul 9 00:40:19.576316511 (375h29m19.45973014s) [seg_pos={0 0}, upload=300262]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 9 00:40:19.685159343 (375h29m19.568572971s) [upload=300262]: Done uploading segments...
Jul 9 00:40:19.685187481 (375h29m19.568601108s) [upload=300262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.401908144s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................X..X..............] duration:
2.82790737s
FAIL (2025-07-09 00:50:32 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
entCommit
Jul 9 00:38:54.141867066 (375h27m54.025280684s) [upload=300248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:54.286588243 (375h27m54.170001881s) [upload=300248]: Done uploading segments...
Jul 9 00:38:54.286633475 (375h27m54.170047105s) [upload=300248]: Upload finished. err: <nil>
Jul 9 00:39:49.853458099 (375h28m49.736871729s) [upload=300262]: Starting upload
Jul 9 00:39:49.853564556 (375h28m49.736978174s) [upload=300262]: Done starting upload
Jul 9 00:39:49.853754759 (375h28m49.737168394s) [upload=300262]: Uploading segments...
Jul 9 00:39:49.854072231 (375h28m49.737485857s) [upload=300262]: Got next segment. Inline: false
Jul 9 00:39:49.854139888 (375h28m49.737553515s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:39:49.854145670 (375h28m49.737559297s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:39:49.964717413 (375h28m49.848131039s) [seg_pos={0 0}, upload=300262]: Begin upload segment...
Jul 9 00:39:49.964772224 (375h28m49.848185839s) [seg_pos={0 0}, upload=300262]: Done begin upload segment.
Jul 9 00:39:49.964909767 (375h28m49.848323393s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965422318 (375h28m49.848835947s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965585955 (375h28m49.848999581s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966217091 (375h28m49.849630727s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966292916 (375h28m49.849706542s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967093655 (375h28m49.85050729s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967169721 (375h28m49.850583348s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967940005 (375h28m49.851353634s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.968588971 (375h28m49.8520026s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969080450 (375h28m49.852494085s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969563775 (375h28m49.852977409s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.970736354 (375h28m49.854149991s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.971229734 (375h28m49.85464336s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.972867679 (375h28m49.856281317s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.978267895 (375h28m49.861681529s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.986076032 (375h28m49.869489662s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:50.595435555 (375h28m50.478849173s) [upload=300262]: Next returned nil segment
Jul 9 00:39:50.595455642 (375h28m50.478869272s) [upload=300262]: Waiting for error group managing segments...
Jul 9 00:39:50.730628340 (375h28m50.614041956s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.731179161 (375h28m50.614592788s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.732951876 (375h28m50.616365493s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.738403976 (375h28m50.621817592s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739011956 (375h28m50.622425573s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739658988 (375h28m50.623072606s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.766298764 (375h28m50.649712392s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.754942777 (375h28m51.638356391s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.990915358 (375h28m51.87432899s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:13.196278807 (375h29m13.079692437s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:19.575930850 (375h29m19.459344476s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576005813 (375h29m19.459419439s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576051734 (375h29m19.459465361s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576066765 (375h29m19.459480398s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576255242 (375h29m19.459668869s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection
Jul 9 00:40:19.576293041 (375h29m19.459706667s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection
Jul 9 00:40:19.576316511 (375h29m19.45973014s) [seg_pos={0 0}, upload=300262]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 9 00:40:19.685159343 (375h29m19.568572971s) [upload=300262]: Done uploading segments...
Jul 9 00:40:19.685187481 (375h29m19.568601108s) [upload=300262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.827945231s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
12 out of 60 is failed (threshold=3) [.XX.X.................X...........XX.X..X..X..XX.X..........] duration:
50.985595976s
FAIL (2025-07-09 00:50:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 50.985638621s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [........XX...X..............X....X..X......X.............XX.] duration:
8.632422668s
FAIL (2025-07-09 00:50:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.632441384s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................................X..X.............] duration:
27.838152049s
FAIL (2025-07-09 00:50:10 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
entCommit
Jul 9 00:38:54.141867066 (375h27m54.025280684s) [upload=300248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:54.286588243 (375h27m54.170001881s) [upload=300248]: Done uploading segments...
Jul 9 00:38:54.286633475 (375h27m54.170047105s) [upload=300248]: Upload finished. err: <nil>
Jul 9 00:39:49.853458099 (375h28m49.736871729s) [upload=300262]: Starting upload
Jul 9 00:39:49.853564556 (375h28m49.736978174s) [upload=300262]: Done starting upload
Jul 9 00:39:49.853754759 (375h28m49.737168394s) [upload=300262]: Uploading segments...
Jul 9 00:39:49.854072231 (375h28m49.737485857s) [upload=300262]: Got next segment. Inline: false
Jul 9 00:39:49.854139888 (375h28m49.737553515s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:39:49.854145670 (375h28m49.737559297s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:39:49.964717413 (375h28m49.848131039s) [seg_pos={0 0}, upload=300262]: Begin upload segment...
Jul 9 00:39:49.964772224 (375h28m49.848185839s) [seg_pos={0 0}, upload=300262]: Done begin upload segment.
Jul 9 00:39:49.964909767 (375h28m49.848323393s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965422318 (375h28m49.848835947s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965585955 (375h28m49.848999581s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966217091 (375h28m49.849630727s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966292916 (375h28m49.849706542s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967093655 (375h28m49.85050729s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967169721 (375h28m49.850583348s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967940005 (375h28m49.851353634s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.968588971 (375h28m49.8520026s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969080450 (375h28m49.852494085s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969563775 (375h28m49.852977409s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.970736354 (375h28m49.854149991s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.971229734 (375h28m49.85464336s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.972867679 (375h28m49.856281317s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.978267895 (375h28m49.861681529s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.986076032 (375h28m49.869489662s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:50.595435555 (375h28m50.478849173s) [upload=300262]: Next returned nil segment
Jul 9 00:39:50.595455642 (375h28m50.478869272s) [upload=300262]: Waiting for error group managing segments...
Jul 9 00:39:50.730628340 (375h28m50.614041956s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.731179161 (375h28m50.614592788s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.732951876 (375h28m50.616365493s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.738403976 (375h28m50.621817592s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739011956 (375h28m50.622425573s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739658988 (375h28m50.623072606s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.766298764 (375h28m50.649712392s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.754942777 (375h28m51.638356391s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.990915358 (375h28m51.87432899s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:13.196278807 (375h29m13.079692437s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:19.575930850 (375h29m19.459344476s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576005813 (375h29m19.459419439s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576051734 (375h29m19.459465361s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576066765 (375h29m19.459480398s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576255242 (375h29m19.459668869s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection
Jul 9 00:40:19.576293041 (375h29m19.459706667s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection
Jul 9 00:40:19.576316511 (375h29m19.45973014s) [seg_pos={0 0}, upload=300262]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 9 00:40:19.685159343 (375h29m19.568572971s) [upload=300262]: Done uploading segments...
Jul 9 00:40:19.685187481 (375h29m19.568601108s) [upload=300262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 27.838217055s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [X........XX...X..............X....X..X......X.............XX] duration:
30.001492301s
FAIL (2025-07-09 00:50:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001519105s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
18.820187128s
FAIL (2025-07-09 00:49:52 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 18.820227271s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
12 out of 60 is failed (threshold=3) [..XX.X.................X...........XX.X..X..X..XX.X.........] duration:
45.116425933s
FAIL (2025-07-09 00:49:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 45.116472542s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................................X..X............] duration:
17.624399322s
FAIL (2025-07-09 00:49:19 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
entCommit
Jul 9 00:38:54.141867066 (375h27m54.025280684s) [upload=300248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:54.286588243 (375h27m54.170001881s) [upload=300248]: Done uploading segments...
Jul 9 00:38:54.286633475 (375h27m54.170047105s) [upload=300248]: Upload finished. err: <nil>
Jul 9 00:39:49.853458099 (375h28m49.736871729s) [upload=300262]: Starting upload
Jul 9 00:39:49.853564556 (375h28m49.736978174s) [upload=300262]: Done starting upload
Jul 9 00:39:49.853754759 (375h28m49.737168394s) [upload=300262]: Uploading segments...
Jul 9 00:39:49.854072231 (375h28m49.737485857s) [upload=300262]: Got next segment. Inline: false
Jul 9 00:39:49.854139888 (375h28m49.737553515s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:39:49.854145670 (375h28m49.737559297s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:39:49.964717413 (375h28m49.848131039s) [seg_pos={0 0}, upload=300262]: Begin upload segment...
Jul 9 00:39:49.964772224 (375h28m49.848185839s) [seg_pos={0 0}, upload=300262]: Done begin upload segment.
Jul 9 00:39:49.964909767 (375h28m49.848323393s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965422318 (375h28m49.848835947s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965585955 (375h28m49.848999581s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966217091 (375h28m49.849630727s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966292916 (375h28m49.849706542s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967093655 (375h28m49.85050729s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967169721 (375h28m49.850583348s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967940005 (375h28m49.851353634s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.968588971 (375h28m49.8520026s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969080450 (375h28m49.852494085s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969563775 (375h28m49.852977409s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.970736354 (375h28m49.854149991s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.971229734 (375h28m49.85464336s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.972867679 (375h28m49.856281317s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.978267895 (375h28m49.861681529s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.986076032 (375h28m49.869489662s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:50.595435555 (375h28m50.478849173s) [upload=300262]: Next returned nil segment
Jul 9 00:39:50.595455642 (375h28m50.478869272s) [upload=300262]: Waiting for error group managing segments...
Jul 9 00:39:50.730628340 (375h28m50.614041956s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.731179161 (375h28m50.614592788s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.732951876 (375h28m50.616365493s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.738403976 (375h28m50.621817592s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739011956 (375h28m50.622425573s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739658988 (375h28m50.623072606s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.766298764 (375h28m50.649712392s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.754942777 (375h28m51.638356391s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.990915358 (375h28m51.87432899s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:13.196278807 (375h29m13.079692437s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:19.575930850 (375h29m19.459344476s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576005813 (375h29m19.459419439s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576051734 (375h29m19.459465361s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576066765 (375h29m19.459480398s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576255242 (375h29m19.459668869s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection
Jul 9 00:40:19.576293041 (375h29m19.459706667s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection
Jul 9 00:40:19.576316511 (375h29m19.45973014s) [seg_pos={0 0}, upload=300262]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 9 00:40:19.685159343 (375h29m19.568572971s) [upload=300262]: Done uploading segments...
Jul 9 00:40:19.685187481 (375h29m19.568601108s) [upload=300262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 17.624433329s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.X........XX...X..............X....X..X......X.............X] duration:
30.001667412s
FAIL (2025-07-09 00:49:13 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001780747s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
23.065112394s
FAIL (2025-07-09 00:48:47 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 23.065149542s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [X..XX.X.................X...........XX.X..X..X..XX.X........] duration:
38.029999988s
FAIL (2025-07-09 00:48:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 38.030030915s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
55.704975085s
FAIL (2025-07-09 00:48:23 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 55.705013884s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..X........XX...X..............X....X..X......X.............] duration:
17.993390436s
FAIL (2025-07-09 00:48:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.994273585s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................................X..X...........] duration:
2.333022177s
FAIL (2025-07-09 00:48:17 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
entCommit
Jul 9 00:38:54.141867066 (375h27m54.025280684s) [upload=300248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:54.286588243 (375h27m54.170001881s) [upload=300248]: Done uploading segments...
Jul 9 00:38:54.286633475 (375h27m54.170047105s) [upload=300248]: Upload finished. err: <nil>
Jul 9 00:39:49.853458099 (375h28m49.736871729s) [upload=300262]: Starting upload
Jul 9 00:39:49.853564556 (375h28m49.736978174s) [upload=300262]: Done starting upload
Jul 9 00:39:49.853754759 (375h28m49.737168394s) [upload=300262]: Uploading segments...
Jul 9 00:39:49.854072231 (375h28m49.737485857s) [upload=300262]: Got next segment. Inline: false
Jul 9 00:39:49.854139888 (375h28m49.737553515s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:39:49.854145670 (375h28m49.737559297s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:39:49.964717413 (375h28m49.848131039s) [seg_pos={0 0}, upload=300262]: Begin upload segment...
Jul 9 00:39:49.964772224 (375h28m49.848185839s) [seg_pos={0 0}, upload=300262]: Done begin upload segment.
Jul 9 00:39:49.964909767 (375h28m49.848323393s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965422318 (375h28m49.848835947s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965585955 (375h28m49.848999581s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966217091 (375h28m49.849630727s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966292916 (375h28m49.849706542s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967093655 (375h28m49.85050729s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967169721 (375h28m49.850583348s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967940005 (375h28m49.851353634s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.968588971 (375h28m49.8520026s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969080450 (375h28m49.852494085s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969563775 (375h28m49.852977409s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.970736354 (375h28m49.854149991s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.971229734 (375h28m49.85464336s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.972867679 (375h28m49.856281317s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.978267895 (375h28m49.861681529s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.986076032 (375h28m49.869489662s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:50.595435555 (375h28m50.478849173s) [upload=300262]: Next returned nil segment
Jul 9 00:39:50.595455642 (375h28m50.478869272s) [upload=300262]: Waiting for error group managing segments...
Jul 9 00:39:50.730628340 (375h28m50.614041956s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.731179161 (375h28m50.614592788s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.732951876 (375h28m50.616365493s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.738403976 (375h28m50.621817592s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739011956 (375h28m50.622425573s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739658988 (375h28m50.623072606s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.766298764 (375h28m50.649712392s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.754942777 (375h28m51.638356391s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.990915358 (375h28m51.87432899s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:13.196278807 (375h29m13.079692437s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:19.575930850 (375h29m19.459344476s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576005813 (375h29m19.459419439s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576051734 (375h29m19.459465361s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576066765 (375h29m19.459480398s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576255242 (375h29m19.459668869s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection
Jul 9 00:40:19.576293041 (375h29m19.459706667s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection
Jul 9 00:40:19.576316511 (375h29m19.45973014s) [seg_pos={0 0}, upload=300262]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 9 00:40:19.685159343 (375h29m19.568572971s) [upload=300262]: Done uploading segments...
Jul 9 00:40:19.685187481 (375h29m19.568601108s) [upload=300262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.333045328s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................................X..X..........] duration:
2.417646064s
FAIL (2025-07-09 00:47:42 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
entCommit
Jul 9 00:38:54.141867066 (375h27m54.025280684s) [upload=300248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:54.286588243 (375h27m54.170001881s) [upload=300248]: Done uploading segments...
Jul 9 00:38:54.286633475 (375h27m54.170047105s) [upload=300248]: Upload finished. err: <nil>
Jul 9 00:39:49.853458099 (375h28m49.736871729s) [upload=300262]: Starting upload
Jul 9 00:39:49.853564556 (375h28m49.736978174s) [upload=300262]: Done starting upload
Jul 9 00:39:49.853754759 (375h28m49.737168394s) [upload=300262]: Uploading segments...
Jul 9 00:39:49.854072231 (375h28m49.737485857s) [upload=300262]: Got next segment. Inline: false
Jul 9 00:39:49.854139888 (375h28m49.737553515s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:39:49.854145670 (375h28m49.737559297s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:39:49.964717413 (375h28m49.848131039s) [seg_pos={0 0}, upload=300262]: Begin upload segment...
Jul 9 00:39:49.964772224 (375h28m49.848185839s) [seg_pos={0 0}, upload=300262]: Done begin upload segment.
Jul 9 00:39:49.964909767 (375h28m49.848323393s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965422318 (375h28m49.848835947s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965585955 (375h28m49.848999581s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966217091 (375h28m49.849630727s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966292916 (375h28m49.849706542s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967093655 (375h28m49.85050729s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967169721 (375h28m49.850583348s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967940005 (375h28m49.851353634s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.968588971 (375h28m49.8520026s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969080450 (375h28m49.852494085s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969563775 (375h28m49.852977409s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.970736354 (375h28m49.854149991s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.971229734 (375h28m49.85464336s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.972867679 (375h28m49.856281317s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.978267895 (375h28m49.861681529s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.986076032 (375h28m49.869489662s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:50.595435555 (375h28m50.478849173s) [upload=300262]: Next returned nil segment
Jul 9 00:39:50.595455642 (375h28m50.478869272s) [upload=300262]: Waiting for error group managing segments...
Jul 9 00:39:50.730628340 (375h28m50.614041956s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.731179161 (375h28m50.614592788s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.732951876 (375h28m50.616365493s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.738403976 (375h28m50.621817592s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739011956 (375h28m50.622425573s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739658988 (375h28m50.623072606s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.766298764 (375h28m50.649712392s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.754942777 (375h28m51.638356391s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.990915358 (375h28m51.87432899s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:13.196278807 (375h29m13.079692437s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:19.575930850 (375h29m19.459344476s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576005813 (375h29m19.459419439s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576051734 (375h29m19.459465361s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576066765 (375h29m19.459480398s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576255242 (375h29m19.459668869s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection
Jul 9 00:40:19.576293041 (375h29m19.459706667s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection
Jul 9 00:40:19.576316511 (375h29m19.45973014s) [seg_pos={0 0}, upload=300262]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 9 00:40:19.685159343 (375h29m19.568572971s) [upload=300262]: Done uploading segments...
Jul 9 00:40:19.685187481 (375h29m19.568601108s) [upload=300262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.417678487s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...X........XX...X..............X....X..X......X............] duration:
14.752988268s
FAIL (2025-07-09 00:47:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.753088628s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
37.790447262s
FAIL (2025-07-09 00:47:28 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 37.790482157s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [.X..XX.X.................X...........XX.X..X..X..XX.X.......] duration:
43.94190979s
FAIL (2025-07-09 00:47:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 43.941932722s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...............................................X..X.........] duration:
6.877030228s
FAIL (2025-07-09 00:47:02 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
entCommit
Jul 9 00:38:54.141867066 (375h27m54.025280684s) [upload=300248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:54.286588243 (375h27m54.170001881s) [upload=300248]: Done uploading segments...
Jul 9 00:38:54.286633475 (375h27m54.170047105s) [upload=300248]: Upload finished. err: <nil>
Jul 9 00:39:49.853458099 (375h28m49.736871729s) [upload=300262]: Starting upload
Jul 9 00:39:49.853564556 (375h28m49.736978174s) [upload=300262]: Done starting upload
Jul 9 00:39:49.853754759 (375h28m49.737168394s) [upload=300262]: Uploading segments...
Jul 9 00:39:49.854072231 (375h28m49.737485857s) [upload=300262]: Got next segment. Inline: false
Jul 9 00:39:49.854139888 (375h28m49.737553515s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:39:49.854145670 (375h28m49.737559297s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:39:49.964717413 (375h28m49.848131039s) [seg_pos={0 0}, upload=300262]: Begin upload segment...
Jul 9 00:39:49.964772224 (375h28m49.848185839s) [seg_pos={0 0}, upload=300262]: Done begin upload segment.
Jul 9 00:39:49.964909767 (375h28m49.848323393s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965422318 (375h28m49.848835947s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965585955 (375h28m49.848999581s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966217091 (375h28m49.849630727s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966292916 (375h28m49.849706542s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967093655 (375h28m49.85050729s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967169721 (375h28m49.850583348s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967940005 (375h28m49.851353634s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.968588971 (375h28m49.8520026s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969080450 (375h28m49.852494085s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969563775 (375h28m49.852977409s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.970736354 (375h28m49.854149991s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.971229734 (375h28m49.85464336s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.972867679 (375h28m49.856281317s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.978267895 (375h28m49.861681529s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.986076032 (375h28m49.869489662s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:50.595435555 (375h28m50.478849173s) [upload=300262]: Next returned nil segment
Jul 9 00:39:50.595455642 (375h28m50.478869272s) [upload=300262]: Waiting for error group managing segments...
Jul 9 00:39:50.730628340 (375h28m50.614041956s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.731179161 (375h28m50.614592788s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.732951876 (375h28m50.616365493s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.738403976 (375h28m50.621817592s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739011956 (375h28m50.622425573s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739658988 (375h28m50.623072606s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.766298764 (375h28m50.649712392s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.754942777 (375h28m51.638356391s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.990915358 (375h28m51.87432899s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:13.196278807 (375h29m13.079692437s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:19.575930850 (375h29m19.459344476s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576005813 (375h29m19.459419439s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576051734 (375h29m19.459465361s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576066765 (375h29m19.459480398s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576255242 (375h29m19.459668869s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection
Jul 9 00:40:19.576293041 (375h29m19.459706667s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection
Jul 9 00:40:19.576316511 (375h29m19.45973014s) [seg_pos={0 0}, upload=300262]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 9 00:40:19.685159343 (375h29m19.568572971s) [upload=300262]: Done uploading segments...
Jul 9 00:40:19.685187481 (375h29m19.568601108s) [upload=300262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 6.877068339s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [....X........XX...X..............X....X..X......X...........] duration:
3.16263609s
FAIL (2025-07-09 00:47:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.16276318s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.....X........XX...X..............X....X..X......X..........] duration:
20.417501605s
FAIL (2025-07-09 00:46:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 20.417529908s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
18.642862368s
FAIL (2025-07-09 00:46:37 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 18.642897527s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [..X..XX.X.................X...........XX.X..X..X..XX.X......] duration:
45.453217621s
FAIL (2025-07-09 00:46:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 45.453236274s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [................................................X..X........] duration:
7.508977744s
FAIL (2025-07-09 00:46:22 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
entCommit
Jul 9 00:38:54.141867066 (375h27m54.025280684s) [upload=300248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:54.286588243 (375h27m54.170001881s) [upload=300248]: Done uploading segments...
Jul 9 00:38:54.286633475 (375h27m54.170047105s) [upload=300248]: Upload finished. err: <nil>
Jul 9 00:39:49.853458099 (375h28m49.736871729s) [upload=300262]: Starting upload
Jul 9 00:39:49.853564556 (375h28m49.736978174s) [upload=300262]: Done starting upload
Jul 9 00:39:49.853754759 (375h28m49.737168394s) [upload=300262]: Uploading segments...
Jul 9 00:39:49.854072231 (375h28m49.737485857s) [upload=300262]: Got next segment. Inline: false
Jul 9 00:39:49.854139888 (375h28m49.737553515s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:39:49.854145670 (375h28m49.737559297s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:39:49.964717413 (375h28m49.848131039s) [seg_pos={0 0}, upload=300262]: Begin upload segment...
Jul 9 00:39:49.964772224 (375h28m49.848185839s) [seg_pos={0 0}, upload=300262]: Done begin upload segment.
Jul 9 00:39:49.964909767 (375h28m49.848323393s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965422318 (375h28m49.848835947s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965585955 (375h28m49.848999581s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966217091 (375h28m49.849630727s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966292916 (375h28m49.849706542s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967093655 (375h28m49.85050729s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967169721 (375h28m49.850583348s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967940005 (375h28m49.851353634s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.968588971 (375h28m49.8520026s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969080450 (375h28m49.852494085s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969563775 (375h28m49.852977409s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.970736354 (375h28m49.854149991s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.971229734 (375h28m49.85464336s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.972867679 (375h28m49.856281317s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.978267895 (375h28m49.861681529s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.986076032 (375h28m49.869489662s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:50.595435555 (375h28m50.478849173s) [upload=300262]: Next returned nil segment
Jul 9 00:39:50.595455642 (375h28m50.478869272s) [upload=300262]: Waiting for error group managing segments...
Jul 9 00:39:50.730628340 (375h28m50.614041956s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.731179161 (375h28m50.614592788s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.732951876 (375h28m50.616365493s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.738403976 (375h28m50.621817592s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739011956 (375h28m50.622425573s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739658988 (375h28m50.623072606s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.766298764 (375h28m50.649712392s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.754942777 (375h28m51.638356391s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.990915358 (375h28m51.87432899s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:13.196278807 (375h29m13.079692437s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:19.575930850 (375h29m19.459344476s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576005813 (375h29m19.459419439s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576051734 (375h29m19.459465361s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576066765 (375h29m19.459480398s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576255242 (375h29m19.459668869s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection
Jul 9 00:40:19.576293041 (375h29m19.459706667s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection
Jul 9 00:40:19.576316511 (375h29m19.45973014s) [seg_pos={0 0}, upload=300262]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 9 00:40:19.685159343 (375h29m19.568572971s) [upload=300262]: Done uploading segments...
Jul 9 00:40:19.685187481 (375h29m19.568601108s) [upload=300262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 7.509006688s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
37.296900038s
FAIL (2025-07-09 00:45:54 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 37.296927328s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.................................................X..X.......] duration:
2.485231802s
FAIL (2025-07-09 00:45:48 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
entCommit
Jul 9 00:38:54.141867066 (375h27m54.025280684s) [upload=300248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:54.286588243 (375h27m54.170001881s) [upload=300248]: Done uploading segments...
Jul 9 00:38:54.286633475 (375h27m54.170047105s) [upload=300248]: Upload finished. err: <nil>
Jul 9 00:39:49.853458099 (375h28m49.736871729s) [upload=300262]: Starting upload
Jul 9 00:39:49.853564556 (375h28m49.736978174s) [upload=300262]: Done starting upload
Jul 9 00:39:49.853754759 (375h28m49.737168394s) [upload=300262]: Uploading segments...
Jul 9 00:39:49.854072231 (375h28m49.737485857s) [upload=300262]: Got next segment. Inline: false
Jul 9 00:39:49.854139888 (375h28m49.737553515s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:39:49.854145670 (375h28m49.737559297s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:39:49.964717413 (375h28m49.848131039s) [seg_pos={0 0}, upload=300262]: Begin upload segment...
Jul 9 00:39:49.964772224 (375h28m49.848185839s) [seg_pos={0 0}, upload=300262]: Done begin upload segment.
Jul 9 00:39:49.964909767 (375h28m49.848323393s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965422318 (375h28m49.848835947s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965585955 (375h28m49.848999581s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966217091 (375h28m49.849630727s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966292916 (375h28m49.849706542s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967093655 (375h28m49.85050729s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967169721 (375h28m49.850583348s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967940005 (375h28m49.851353634s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.968588971 (375h28m49.8520026s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969080450 (375h28m49.852494085s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969563775 (375h28m49.852977409s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.970736354 (375h28m49.854149991s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.971229734 (375h28m49.85464336s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.972867679 (375h28m49.856281317s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.978267895 (375h28m49.861681529s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.986076032 (375h28m49.869489662s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:50.595435555 (375h28m50.478849173s) [upload=300262]: Next returned nil segment
Jul 9 00:39:50.595455642 (375h28m50.478869272s) [upload=300262]: Waiting for error group managing segments...
Jul 9 00:39:50.730628340 (375h28m50.614041956s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.731179161 (375h28m50.614592788s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.732951876 (375h28m50.616365493s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.738403976 (375h28m50.621817592s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739011956 (375h28m50.622425573s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739658988 (375h28m50.623072606s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.766298764 (375h28m50.649712392s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.754942777 (375h28m51.638356391s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.990915358 (375h28m51.87432899s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:13.196278807 (375h29m13.079692437s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:19.575930850 (375h29m19.459344476s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576005813 (375h29m19.459419439s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576051734 (375h29m19.459465361s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576066765 (375h29m19.459480398s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576255242 (375h29m19.459668869s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection
Jul 9 00:40:19.576293041 (375h29m19.459706667s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection
Jul 9 00:40:19.576316511 (375h29m19.45973014s) [seg_pos={0 0}, upload=300262]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 9 00:40:19.685159343 (375h29m19.568572971s) [upload=300262]: Done uploading segments...
Jul 9 00:40:19.685187481 (375h29m19.568601108s) [upload=300262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.485264651s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [...X..XX.X.................X...........XX.X..X..X..XX.X.....] duration:
56.339333541s
FAIL (2025-07-09 00:45:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 56.339356215s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [......X........XX...X..............X....X..X......X.........] duration:
17.659761176s
FAIL (2025-07-09 00:45:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 17.659797238s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
33.319343585s
FAIL (2025-07-09 00:45:11 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 33.319367024s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
13 out of 60 is failed (threshold=3) [....X..XX.X.................X...........XX.X..X..X..XX.X....] duration:
54.893385811s
FAIL (2025-07-09 00:44:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 54.893400127s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
37.709735787s
FAIL (2025-07-09 00:44:38 UTC)
orbiter warning:
======ERROR======
uplink: metainfo: metaclient: context deadline exceeded
======LOGS======
3980 (375h26m3.9562776s) [upload=300223]: Starting upload
Jul 9 00:37:04.072968784 (375h26m3.956382412s) [upload=300223]: Done starting upload
Jul 9 00:37:04.073087735 (375h26m3.956501365s) [upload=300223]: Uploading segments...
Jul 9 00:37:04.073377778 (375h26m3.956791396s) [upload=300223]: Got next segment. Inline: false
Jul 9 00:37:04.073430757 (375h26m3.956844388s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:37:04.073437151 (375h26m3.956850769s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:37:04.304692457 (375h26m4.18810608s) [seg_pos={0 0}, upload=300223]: Begin upload segment...
Jul 9 00:37:04.304761975 (375h26m4.188175587s) [seg_pos={0 0}, upload=300223]: Done begin upload segment.
Jul 9 00:37:04.304833908 (375h26m4.18824754s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305365539 (375h26m4.18877917s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305738725 (375h26m4.189152359s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.305907885 (375h26m4.189321515s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306301693 (375h26m4.189715324s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.306679055 (375h26m4.190092689s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307048455 (375h26m4.190462086s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.307412956 (375h26m4.19082659s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.308062976 (375h26m4.191476609s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.309126547 (375h26m4.192540184s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.310676166 (375h26m4.194089804s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311146138 (375h26m4.19455977s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311543981 (375h26m4.194957595s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.311932868 (375h26m4.195346498s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312324028 (375h26m4.195737661s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.312707561 (375h26m4.196121192s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313124992 (375h26m4.19653862s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.313485189 (375h26m4.196898821s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Uploading piece...
Jul 9 00:37:04.893020764 (375h26m4.776434382s) [upload=300223]: Next returned nil segment
Jul 9 00:37:04.893957299 (375h26m4.777370932s) [upload=300223]: Waiting for error group managing segments...
Jul 9 00:37:05.037354849 (375h26m4.920768466s) [noise=true , address=77.90.31.10:4547 , piece_id=V3YTAVFJL7UPZYPJXQL3VWHNZHTQZBQ3P46BEULJGRT4MRI5OFTQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.038444983 (375h26m4.921858597s) [noise=true , address=77.90.31.16:4765 , piece_id=FMYOKA3GUMYAQJFLKULKN37AGXAQNMDUXVZTVA5M7UFM76XLQPXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.039356590 (375h26m4.922770207s) [noise=true , address=77.90.31.16:4763 , piece_id=PTKGN4O4BD5EZUFJ7DJYNEPYAB4RAXUCPZI7ZPKMGAUEXCA2Y7AA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.047559125 (375h26m4.930972742s) [noise=true , address=77.90.31.16:4736 , piece_id=MM6DFSLH64WYDRQUNNTABPUIQF7UI6RWIJRUELQGU33RDHEAXMRA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.053248590 (375h26m4.93666221s) [noise=true , address=77.90.31.17:4795 , piece_id=ABVYKQWDKVSSLA2GQWDSNXMIBFZRN6YLISO6VRANUA6HAEQAHYFQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.054452827 (375h26m4.937866447s) [noise=true , address=77.90.31.15:4727 , piece_id=IDT7ZUGMUHZQNFGIVHAKD2A7Z6I4726HZHDR2UZMQOBH7AOB27TQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:05.889326616 (375h26m5.772740232s) [noise=true , address=79.127.219.48:4076 , piece_id=KXP6PMGN5ZI2SIQKKXL23ENORCC274V6YRU57AVAPAOO5ZCZUBIQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:29.628225928 (375h26m29.511639541s) [noise=true , address=104.36.82.6:4898 , piece_id=YFHP6ZZY3FRU7Y3GWU5VDJ6YX3BGFZUR2WBDYVI24MZCE5CROJRQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:30.256578837 (375h26m30.139992453s) [noise=true , address=104.36.82.42:4817 , piece_id=CN3V3YEEJLS3U7ZCDOKQQRB5TZE2HKOJYF4NJAJJ2CAMXME7Q2ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:32.331498448 (375h26m32.214912065s) [noise=true , address=104.36.82.38:4708 , piece_id=MWDSQCSF2KOZFDEXUWK7BS6G2MYG4EIVOZDDZQPLDGSVXOLVJG7A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:39.883025841 (375h26m39.766439475s) [noise=true , address=104.36.82.35:4614 , piece_id=IS5NVS4QT3TNFR744BJXYYHSNBGNS425AAJLGHJYTYJQ4R44NB2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:51.499841545 (375h26m51.383255165s) [noise=true , address=104.36.82.60:4368 , piece_id=5C5C6KCHEC2ZSIBNAWRBEV442DKVID6ILQ7JA6DFFGB6BBV2EVXA, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:37:55.982236621 (375h26m55.86565024s) [noise=true , address=104.36.82.10:4091 , piece_id=2CZWVKEU2J2IEC3M4JTU5G2TZ2P2ERRLMID2CZR5X4L7G5WMZRYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607812826 (375h27m3.49122645s) [noise=true , address=104.36.82.44:4882 , piece_id=46O66XTAJO5M5EIN75SIZ7HAFK6C3P5VNH2R3BN3HXRLBPHQOV2Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: <nil>
Jul 9 00:38:03.607853389 (375h27m3.491267022s) [seg_pos={0 0}, upload=300223]: Segment reached optimal threshold of 14 pieces.
Jul 9 00:38:03.608006922 (375h27m3.491420552s) [noise=true , address=104.36.82.19:4427 , piece_id=LGLX2VGSEE6SVNRZSKX46A5WGT3SLKH2EO6VM3SYJXRZX5V5GP6Q, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1NdaXHU5WMvCtca7R6XRycKQGBQio4cikujGTu8TwnyPcGf7uG): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608045600 (375h27m3.491459233s) [noise=true , address=104.36.82.4:4809 , piece_id=WYLOHB5UOZWNLSPDSZ2GNFVQGG5HD4OIKKZGPF5ZFMCHU45MGNYQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608087309 (375h27m3.491500937s) [noise=true , address=104.36.82.22:4540 , piece_id=NR7IGPRLKNCRVYDVMGTMAPHHAV5W7VHEMCUDZ3C7MX372WH765ZQ, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RY6XGL5MJYc2idL6Np7DS3U9pCQGDhuEEqguY8AgjasjE4hiZ): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608191697 (375h27m3.491605324s) [noise=true , address=104.36.82.36:4631 , piece_id=WDDGALSSDOMMI2SRRAXKWD3SD7WI7PCA4L33Q2WP3IZ7UTMH2R4A, seg_pos={0 0}, upload=300223]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T6FqqMunZ46QoxhHVWsn8NinZGWv1Yo7rNHSiK6s9hhuQtS6Y): protocol: expected piece hash; context canceled; EOF
Jul 9 00:38:03.608213412 (375h27m3.49162703s) [seg_pos={0 0}, upload=300223]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jul 9 00:38:03.608310462 (375h27m3.49172408s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jul 9 00:38:03.608316810 (375h27m3.491730438s) [upload=300223]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:03.761052568 (375h27m3.6444662s) [upload=300223]: Done uploading segments...
Jul 9 00:38:03.761076757 (375h27m3.644490378s) [upload=300223]: Upload finished. err: <nil>
Duration: 37.709763906s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [.......X........XX...X..............X....X..X......X........] duration:
5.250548778s
FAIL (2025-07-09 00:44:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.250576584s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................................X..X......] duration:
2.325436803s
FAIL (2025-07-09 00:44:36 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 10)
======LOGS======
entCommit
Jul 9 00:38:54.141867066 (375h27m54.025280684s) [upload=300248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 9 00:38:54.286588243 (375h27m54.170001881s) [upload=300248]: Done uploading segments...
Jul 9 00:38:54.286633475 (375h27m54.170047105s) [upload=300248]: Upload finished. err: <nil>
Jul 9 00:39:49.853458099 (375h28m49.736871729s) [upload=300262]: Starting upload
Jul 9 00:39:49.853564556 (375h28m49.736978174s) [upload=300262]: Done starting upload
Jul 9 00:39:49.853754759 (375h28m49.737168394s) [upload=300262]: Uploading segments...
Jul 9 00:39:49.854072231 (375h28m49.737485857s) [upload=300262]: Got next segment. Inline: false
Jul 9 00:39:49.854139888 (375h28m49.737553515s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 9 00:39:49.854145670 (375h28m49.737559297s) [upload=300262]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 9 00:39:49.964717413 (375h28m49.848131039s) [seg_pos={0 0}, upload=300262]: Begin upload segment...
Jul 9 00:39:49.964772224 (375h28m49.848185839s) [seg_pos={0 0}, upload=300262]: Done begin upload segment.
Jul 9 00:39:49.964909767 (375h28m49.848323393s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965422318 (375h28m49.848835947s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.965585955 (375h28m49.848999581s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966217091 (375h28m49.849630727s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.966292916 (375h28m49.849706542s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967093655 (375h28m49.85050729s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967169721 (375h28m49.850583348s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.967940005 (375h28m49.851353634s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.968588971 (375h28m49.8520026s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969080450 (375h28m49.852494085s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.969563775 (375h28m49.852977409s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.970736354 (375h28m49.854149991s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.971229734 (375h28m49.85464336s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.972867679 (375h28m49.856281317s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.978267895 (375h28m49.861681529s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:49.986076032 (375h28m49.869489662s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Uploading piece...
Jul 9 00:39:50.595435555 (375h28m50.478849173s) [upload=300262]: Next returned nil segment
Jul 9 00:39:50.595455642 (375h28m50.478869272s) [upload=300262]: Waiting for error group managing segments...
Jul 9 00:39:50.730628340 (375h28m50.614041956s) [noise=true , address=45.91.116.4:4938 , piece_id=OTSXKKNFPSNIANVVIRPDRQEFFDWI27D2JNDGPYDI3QIECWKNJVVA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.731179161 (375h28m50.614592788s) [noise=true , address=45.91.116.13:4153 , piece_id=EBVIZJFQDVIOWT7LM24W5JL36AS2QIOOL4BE7U2MWAK4NUMS763Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.732951876 (375h28m50.616365493s) [noise=true , address=45.91.116.11:4119 , piece_id=VVCKJQ525IGLIAUI2QHK6IVTF6CUSVRLATUN52QMBKLJYSZCTENQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.738403976 (375h28m50.621817592s) [noise=true , address=45.91.116.13:4154 , piece_id=L6UICK5NFWW7X7BW7JM6PPAWRQL5RUXV7UDAOSGRDPMR5MILCHEA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739011956 (375h28m50.622425573s) [noise=true , address=45.91.116.8:4045 , piece_id=QJNLPTWM7SKNXSVXRPVYXQJDI23YZ5I4B4SQRYGYWOFW27MWWMWA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.739658988 (375h28m50.623072606s) [noise=true , address=45.91.116.7:4010 , piece_id=LRVYKKDEGXKZJKVX3AFCBSA4MFQWIOZROXBHEY4RC2TFBWMU4Y7Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:50.766298764 (375h28m50.649712392s) [noise=true , address=77.90.31.21:4879 , piece_id=UUSCN7CIUVOMFWISVIIPUCP55VVLJD73FOE44KUWGDX4GCFHB7FQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.754942777 (375h28m51.638356391s) [noise=true , address=104.36.82.16:4293 , piece_id=C4B5T24WIBRTLCYICMPMYJCVSOPAP3VY22AHWUKGGU5T4A4DMWRQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:39:51.990915358 (375h28m51.87432899s) [noise=true , address=104.36.82.16:4295 , piece_id=H277ZJJ3PJ4JKGGXGSCQMCIFBS6RH4EOKIR6G5GSDSNUAYSW6RZA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:13.196278807 (375h29m13.079692437s) [noise=true , address=104.36.82.5:4837 , piece_id=GFLNL2X2E74H4EPHFZZMQ7QELCNUZ45L7QYLRWJXGVXZT2U3KGFA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: <nil>
Jul 9 00:40:19.575930850 (375h29m19.459344476s) [noise=true , address=104.36.82.15:4259 , piece_id=SAFMGMQWVIL2ONP6KBUHVBVRNL3LT7WUDH24SGNWB745D62X6QXQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576005813 (375h29m19.459419439s) [noise=true , address=104.36.82.2:4730 , piece_id=5XR73Z34ZYCGLRYV2ZQWCNCNKX5I7P5MYXT254ZVEHSVPIZ5RGCA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576051734 (375h29m19.459465361s) [noise=true , address=104.36.82.4:4804 , piece_id=5G2JOP4O2EHZVIDIOJBELVH54XPRHSQFXEDKNSD4K4CO4AJYJKFQ, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576066765 (375h29m19.459480398s) [noise=true , address=104.36.82.13:4185 , piece_id=CFDK2QCRSTTHZ4IDXF4Q346AFBDG2KTWOLXAKZX4JWMX7JR754CA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: expected piece hash; context deadline exceeded; EOF
Jul 9 00:40:19.576255242 (375h29m19.459668869s) [noise=true , address=104.36.82.3:4767 , piece_id=XT74I2XQGL56SLXVHN6TLIXLS7LW3VE5H45VE5ZDCTOSBIV5AX2Q, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:38824->104.36.82.3:4767: use of closed network connection
Jul 9 00:40:19.576293041 (375h29m19.459706667s) [noise=true , address=104.36.82.8:4971 , piece_id=ZGDLV2DBTJODCRLQT3G6ZU3SADRENQXMTYNPNX2NMZBAXOD4SWNA, seg_pos={0 0}, upload=300262]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:37934->104.36.82.8:4971: use of closed network connection
Jul 9 00:40:19.576316511 (375h29m19.45973014s) [seg_pos={0 0}, upload=300262]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil>
Jul 9 00:40:19.685159343 (375h29m19.568572971s) [upload=300262]: Done uploading segments...
Jul 9 00:40:19.685187481 (375h29m19.568601108s) [upload=300262]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10)
Duration: 2.325466996s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [........X........XX...X..............X....X..X......X.......] duration:
19.005383956s
FAIL (2025-07-09 00:44:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 19.005424385s
edge
gateway
us-select-1
|