Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................X....................X.........X............] duration:
2.768016745s
FAIL (2025-08-09 03:57:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.768031839s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
6 out of 60 is failed (threshold=3) [...................................X..XXX.....X.X...........] duration:
306.265129ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-09 03:56:50 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 306.292927ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [.......................X...................X..........X.....] duration:
1.037209318s
FAIL (2025-08-09 03:56:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.03723738s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.....................X............X.........................] duration:
848.90701ms
FAIL (2025-08-09 03:56:38 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:38:21.391404361 (10h12m43.350501607s) [upload=10580]: Starting upload Aug 9 03:38:21.391499574 (10h12m43.350596824s) [upload=10580]: Done starting upload Aug 9 03:38:21.391580214 (10h12m43.35067745s) [upload=10580]: Uploading segments... Aug 9 03:38:21.391626103 (10h12m43.350723357s) [upload=10580]: Got next segment. Inline: true Aug 9 03:38:21.391640756 (10h12m43.350738001s) [upload=10580]: Waiting for error group managing segments... Aug 9 03:38:21.391684974 (10h12m43.350782221s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:38:21.391714671 (10h12m43.350811918s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 9 03:38:21.391718801 (10h12m43.35081605s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 9 03:38:21.563255973 (10h12m43.522353225s) [upload=10580]: Done uploading segments... Aug 9 03:38:21.563291106 (10h12m43.522388342s) [upload=10580]: Upload finished. err: metaclient: Too Many Requests
Duration: 848.957982ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [..................................X.......................X.] duration:
2.344672169s
FAIL (2025-08-09 03:56:34 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M
======LOGS======
Aug 9 03:55:22.143759676 (10h29m44.10285692s) [upload=10871]: Starting upload
Aug 9 03:55:22.143812813 (10h29m44.102910057s) [upload=10871]: Done starting upload
Aug 9 03:55:22.143932181 (10h29m44.103029427s) [upload=10871]: Uploading segments...
Aug 9 03:55:22.144063992 (10h29m44.103161237s) [upload=10871]: Got next segment. Inline: false
Aug 9 03:55:22.144103663 (10h29m44.103200909s) [upload=10871]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 9 03:55:22.144108033 (10h29m44.103205279s) [upload=10871]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 9 03:55:22.344140014 (10h29m44.303237259s) [seg_pos={0 0}, upload=10871]: Begin upload segment...
Aug 9 03:55:22.344190230 (10h29m44.303287465s) [seg_pos={0 0}, upload=10871]: Done begin upload segment.
Aug 9 03:55:22.344241693 (10h29m44.303338943s) [noise=true , address=104.36.82.47:4533 , piece_id=SR7G6S2Z3AGRWND572QYBPMKDNRK3MD6QRUMBDYLRM2ZYVNFVUTQ, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.344712874 (10h29m44.303810123s) [noise=true , address=104.36.82.12:4158 , piece_id=NDGI2BLPE4IMV2JWEZ5AXPHRBT6MSME4UNNGYDOKDOIA6PYSS3TA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.345037423 (10h29m44.304134668s) [noise=true , address=104.36.82.23:4458 , piece_id=5R5O366APTWEUDTZG5GWTSDBNX5S53VZPO2K2U3ZD45OEXBSPHUA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.345359148 (10h29m44.304456399s) [noise=true , address=104.36.82.16:4319 , piece_id=JZKTOUWZW7SWGC3737OGKA5R2V64ANU5ZIXNMLDA4GKCG77Y55HA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.345744398 (10h29m44.304841644s) [noise=true , address=104.36.82.45:4913 , piece_id=A53X5FREON2B3TPU4OJA6GY7743MG63MPW6KJEQ4ICMJEV5QZMYQ, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.346149512 (10h29m44.305246803s) [noise=true , address=104.36.82.59:4323 , piece_id=YVKZVOU4QTUNNHC3W3UUF3KISTPJ3HMU5QIGGFOPFRLS2R3XEF3Q, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.346393756 (10h29m44.305491013s) [noise=true , address=104.36.82.39:4739 , piece_id=R3OAUKXTNBEEZMXX3CJDT2CASE3HLBXNPKD4EGRRFFFF7EGTQ2YQ, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.346552983 (10h29m44.305650229s) [noise=true , address=45.91.116.14:4200 , piece_id=3NK2RKJPXS4UAJXY55I2LHQRX7D3ZPKWFG3SMSV4EZ7MHUROSGRA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.346804907 (10h29m44.30590216s) [noise=true , address=104.36.82.45:4907 , piece_id=BP2W2L76MWDS6W2PQRPK2P3COOD75G75D37ZUGFCY2NOWXBSZGEQ, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.346863880 (10h29m44.305961132s) [noise=true , address=104.36.82.46:4482 , piece_id=MJM7TS3NQTQJ2D4SMYDGDS2FX3AUICTMXSYJO2LE4F3F74CFBOAQ, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.347094395 (10h29m44.306191643s) [noise=true , address=104.36.82.15:4267 , piece_id=6QVBPNZGKNZEH35LRTWXLMZIVNYSUXFLGF4UXA7VOBUUI4LHRSLQ, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.347154723 (10h29m44.306251977s) [noise=true , address=104.36.82.2:4759 , piece_id=B35NJJJ7E6ULFD5OGVDMH4OIC77IGELIRP7AUAJWOYJAGKPDNWRA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.347352448 (10h29m44.306449697s) [noise=true , address=104.36.82.44:4872 , piece_id=ZDQMVJGYPL4R7H2MSYDPVGLKN4B7FG3SCGVZMDU5MFLVISQ3DC6Q, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.347443699 (10h29m44.306540953s) [noise=true , address=104.36.82.58:4296 , piece_id=AGVDDRLZXAAXATA72TY76JNH7P6J6RSFRZFCI7FNT6SSEBNQ6HAA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.347752140 (10h29m44.30684939s) [noise=true , address=104.36.82.60:4359 , piece_id=2DZPFT7KW2BGOQZXKLQM64FS4Q7QJQPQUNTDMRPVKV5JU3KQN4MQ, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.348045599 (10h29m44.307142858s) [noise=true , address=104.36.82.38:4720 , piece_id=6BMWQEPTWJXMPD5VYOCHEERBK374TUXUNUFEH4W427CGZU33Z4CA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.348372798 (10h29m44.307470046s) [noise=true , address=104.36.82.21:4496 , piece_id=OPJLKLLJ7SUYZ4BZ325AANRWXDQJM6JPBSC6JHT3WTI5SGVRHQNA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.348673074 (10h29m44.30777033s) [noise=true , address=64.49.13.140:4511 , piece_id=T7H2CXRTLLZOTJTPVD6BYSVSPWEGB4Y3C2VCIZMZGX5F3RKRYNLA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.911067156 (10h29m44.870164401s) [upload=10871]: Next returned nil segment
Aug 9 03:55:22.911084172 (10h29m44.870181416s) [upload=10871]: Waiting for error group managing segments...
Aug 9 03:55:23.048261951 (10h29m45.007359196s) [noise=true , address=45.91.116.14:4200 , piece_id=3NK2RKJPXS4UAJXY55I2LHQRX7D3ZPKWFG3SMSV4EZ7MHUROSGRA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.064939046 (10h29m45.024036292s) [noise=true , address=104.36.82.60:4359 , piece_id=2DZPFT7KW2BGOQZXKLQM64FS4Q7QJQPQUNTDMRPVKV5JU3KQN4MQ, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.066143838 (10h29m45.025241084s) [noise=true , address=104.36.82.47:4533 , piece_id=SR7G6S2Z3AGRWND572QYBPMKDNRK3MD6QRUMBDYLRM2ZYVNFVUTQ, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.068805794 (10h29m45.027903038s) [noise=true , address=104.36.82.59:4323 , piece_id=YVKZVOU4QTUNNHC3W3UUF3KISTPJ3HMU5QIGGFOPFRLS2R3XEF3Q, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.074735109 (10h29m45.03383234s) [noise=true , address=104.36.82.45:4913 , piece_id=A53X5FREON2B3TPU4OJA6GY7743MG63MPW6KJEQ4ICMJEV5QZMYQ, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.076585738 (10h29m45.035682976s) [noise=true , address=104.36.82.15:4267 , piece_id=6QVBPNZGKNZEH35LRTWXLMZIVNYSUXFLGF4UXA7VOBUUI4LHRSLQ, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.078307083 (10h29m45.037404316s) [noise=true , address=104.36.82.44:4872 , piece_id=ZDQMVJGYPL4R7H2MSYDPVGLKN4B7FG3SCGVZMDU5MFLVISQ3DC6Q, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.080903255 (10h29m45.040000501s) [noise=true , address=104.36.82.58:4296 , piece_id=AGVDDRLZXAAXATA72TY76JNH7P6J6RSFRZFCI7FNT6SSEBNQ6HAA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.084965825 (10h29m45.044063057s) [noise=true , address=104.36.82.2:4759 , piece_id=B35NJJJ7E6ULFD5OGVDMH4OIC77IGELIRP7AUAJWOYJAGKPDNWRA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.086897888 (10h29m45.045995124s) [noise=true , address=104.36.82.45:4907 , piece_id=BP2W2L76MWDS6W2PQRPK2P3COOD75G75D37ZUGFCY2NOWXBSZGEQ, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.088234355 (10h29m45.047331591s) [noise=true , address=104.36.82.46:4482 , piece_id=MJM7TS3NQTQJ2D4SMYDGDS2FX3AUICTMXSYJO2LE4F3F74CFBOAQ, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.090326389 (10h29m45.049423623s) [noise=true , address=104.36.82.12:4158 , piece_id=NDGI2BLPE4IMV2JWEZ5AXPHRBT6MSME4UNNGYDOKDOIA6PYSS3TA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.096118399 (10h29m45.055215633s) [noise=true , address=104.36.82.23:4458 , piece_id=5R5O366APTWEUDTZG5GWTSDBNX5S53VZPO2K2U3ZD45OEXBSPHUA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.096577595 (10h29m45.05567483s) [noise=true , address=104.36.82.38:4720 , piece_id=6BMWQEPTWJXMPD5VYOCHEERBK374TUXUNUFEH4W427CGZU33Z4CA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.096630675 (10h29m45.05572792s) [seg_pos={0 0}, upload=10871]: Segment reached optimal threshold of 14 pieces.
Aug 9 03:55:23.096843965 (10h29m45.055941203s) [noise=true , address=104.36.82.21:4496 , piece_id=OPJLKLLJ7SUYZ4BZ325AANRWXDQJM6JPBSC6JHT3WTI5SGVRHQNA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w): protocol: expected piece hash; context canceled; EOF
Aug 9 03:55:23.097165055 (10h29m45.056262294s) [noise=true , address=104.36.82.16:4319 , piece_id=JZKTOUWZW7SWGC3737OGKA5R2V64ANU5ZIXNMLDA4GKCG77Y55HA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hQTG2DzecaoN1Na2mY3JJqNj2qxtJDQkhMYmRmd9hve8D7T9C): protocol: expected piece hash; context canceled; EOF
Aug 9 03:55:23.097248518 (10h29m45.056345754s) [noise=true , address=64.49.13.140:4511 , piece_id=T7H2CXRTLLZOTJTPVD6BYSVSPWEGB4Y3C2VCIZMZGX5F3RKRYNLA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F79nPRFzy9hjCq5KYvLy6bZERw9cEvczCN2o66Cf6ccKEpkAk): protocol: write tcp 139.59.144.22:59802->64.49.13.140:4511: use of closed network connection; write tcp 139.59.144.22:59802->64.49.13.140:4511: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59802->64.49.13.140:4511: use of closed network connection
Aug 9 03:55:23.097330100 (10h29m45.056427345s) [noise=true , address=104.36.82.39:4739 , piece_id=R3OAUKXTNBEEZMXX3CJDT2CASE3HLBXNPKD4EGRRFFFF7EGTQ2YQ, seg_pos={0 0}, upload=10871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A4nWQsS43tSv3atCnj6CB2ZGKFxQUHJp8a7ftP9y1XumeFk33): protocol: expected piece hash; context canceled; EOF
Aug 9 03:55:23.097345526 (10h29m45.056442772s) [seg_pos={0 0}, upload=10871]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 9 03:55:23.097442993 (10h29m45.056540238s) [upload=10871]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 9 03:55:23.097448165 (10h29m45.056545413s) [upload=10871]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 9 03:55:23.258254104 (10h29m45.21735135s) [upload=10871]: Done uploading segments...
Aug 9 03:55:23.258294964 (10h29m45.217392209s) [upload=10871]: Upload finished. err: <nil>
Duration: 2.34470439s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................X....................X.........X...........] duration:
2.85936221s
FAIL (2025-08-09 03:56:20 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.859566922s
edge
gateway
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
6 out of 60 is failed (threshold=3) [....................................X..XXX.....X.X..........] duration:
306.121082ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-09 03:56:16 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 306.220375ms
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [......................X............X........................] duration:
874.642858ms
FAIL (2025-08-09 03:56:10 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:38:21.391404361 (10h12m43.350501607s) [upload=10580]: Starting upload Aug 9 03:38:21.391499574 (10h12m43.350596824s) [upload=10580]: Done starting upload Aug 9 03:38:21.391580214 (10h12m43.35067745s) [upload=10580]: Uploading segments... Aug 9 03:38:21.391626103 (10h12m43.350723357s) [upload=10580]: Got next segment. Inline: true Aug 9 03:38:21.391640756 (10h12m43.350738001s) [upload=10580]: Waiting for error group managing segments... Aug 9 03:38:21.391684974 (10h12m43.350782221s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:38:21.391714671 (10h12m43.350811918s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 9 03:38:21.391718801 (10h12m43.35081605s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 9 03:38:21.563255973 (10h12m43.522353225s) [upload=10580]: Done uploading segments... Aug 9 03:38:21.563291106 (10h12m43.522388342s) [upload=10580]: Upload finished. err: metaclient: Too Many Requests
Duration: 874.671734ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.......................X............X.......................] duration:
988.936716ms
FAIL (2025-08-09 03:55:48 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:38:21.391404361 (10h12m43.350501607s) [upload=10580]: Starting upload Aug 9 03:38:21.391499574 (10h12m43.350596824s) [upload=10580]: Done starting upload Aug 9 03:38:21.391580214 (10h12m43.35067745s) [upload=10580]: Uploading segments... Aug 9 03:38:21.391626103 (10h12m43.350723357s) [upload=10580]: Got next segment. Inline: true Aug 9 03:38:21.391640756 (10h12m43.350738001s) [upload=10580]: Waiting for error group managing segments... Aug 9 03:38:21.391684974 (10h12m43.350782221s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:38:21.391714671 (10h12m43.350811918s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 9 03:38:21.391718801 (10h12m43.35081605s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 9 03:38:21.563255973 (10h12m43.522353225s) [upload=10580]: Done uploading segments... Aug 9 03:38:21.563291106 (10h12m43.522388342s) [upload=10580]: Upload finished. err: metaclient: Too Many Requests
Duration: 988.990085ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [........................X...................X..........X....] duration:
978.640693ms
FAIL (2025-08-09 03:55:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 978.66091ms
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
306.523174ms
FAIL (2025-08-09 03:55:32 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 306.549292ms
edge
authservice
http
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................X....................X.........X..........] duration:
3.814213647s
FAIL (2025-08-09 03:55:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.81423568s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
2 out of 60 is failed (threshold=3) [...................................X.......................X] duration:
1.598806353s
FAIL (2025-08-09 03:55:23 UTC)
orbiter warning:
operation: ListObjects
======ERROR======
unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/4M
======LOGS======
Aug 9 03:55:22.143759676 (10h29m44.10285692s) [upload=10871]: Starting upload
Aug 9 03:55:22.143812813 (10h29m44.102910057s) [upload=10871]: Done starting upload
Aug 9 03:55:22.143932181 (10h29m44.103029427s) [upload=10871]: Uploading segments...
Aug 9 03:55:22.144063992 (10h29m44.103161237s) [upload=10871]: Got next segment. Inline: false
Aug 9 03:55:22.144103663 (10h29m44.103200909s) [upload=10871]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Aug 9 03:55:22.144108033 (10h29m44.103205279s) [upload=10871]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Aug 9 03:55:22.344140014 (10h29m44.303237259s) [seg_pos={0 0}, upload=10871]: Begin upload segment...
Aug 9 03:55:22.344190230 (10h29m44.303287465s) [seg_pos={0 0}, upload=10871]: Done begin upload segment.
Aug 9 03:55:22.344241693 (10h29m44.303338943s) [noise=true , address=104.36.82.47:4533 , piece_id=SR7G6S2Z3AGRWND572QYBPMKDNRK3MD6QRUMBDYLRM2ZYVNFVUTQ, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.344712874 (10h29m44.303810123s) [noise=true , address=104.36.82.12:4158 , piece_id=NDGI2BLPE4IMV2JWEZ5AXPHRBT6MSME4UNNGYDOKDOIA6PYSS3TA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.345037423 (10h29m44.304134668s) [noise=true , address=104.36.82.23:4458 , piece_id=5R5O366APTWEUDTZG5GWTSDBNX5S53VZPO2K2U3ZD45OEXBSPHUA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.345359148 (10h29m44.304456399s) [noise=true , address=104.36.82.16:4319 , piece_id=JZKTOUWZW7SWGC3737OGKA5R2V64ANU5ZIXNMLDA4GKCG77Y55HA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.345744398 (10h29m44.304841644s) [noise=true , address=104.36.82.45:4913 , piece_id=A53X5FREON2B3TPU4OJA6GY7743MG63MPW6KJEQ4ICMJEV5QZMYQ, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.346149512 (10h29m44.305246803s) [noise=true , address=104.36.82.59:4323 , piece_id=YVKZVOU4QTUNNHC3W3UUF3KISTPJ3HMU5QIGGFOPFRLS2R3XEF3Q, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.346393756 (10h29m44.305491013s) [noise=true , address=104.36.82.39:4739 , piece_id=R3OAUKXTNBEEZMXX3CJDT2CASE3HLBXNPKD4EGRRFFFF7EGTQ2YQ, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.346552983 (10h29m44.305650229s) [noise=true , address=45.91.116.14:4200 , piece_id=3NK2RKJPXS4UAJXY55I2LHQRX7D3ZPKWFG3SMSV4EZ7MHUROSGRA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.346804907 (10h29m44.30590216s) [noise=true , address=104.36.82.45:4907 , piece_id=BP2W2L76MWDS6W2PQRPK2P3COOD75G75D37ZUGFCY2NOWXBSZGEQ, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.346863880 (10h29m44.305961132s) [noise=true , address=104.36.82.46:4482 , piece_id=MJM7TS3NQTQJ2D4SMYDGDS2FX3AUICTMXSYJO2LE4F3F74CFBOAQ, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.347094395 (10h29m44.306191643s) [noise=true , address=104.36.82.15:4267 , piece_id=6QVBPNZGKNZEH35LRTWXLMZIVNYSUXFLGF4UXA7VOBUUI4LHRSLQ, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.347154723 (10h29m44.306251977s) [noise=true , address=104.36.82.2:4759 , piece_id=B35NJJJ7E6ULFD5OGVDMH4OIC77IGELIRP7AUAJWOYJAGKPDNWRA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.347352448 (10h29m44.306449697s) [noise=true , address=104.36.82.44:4872 , piece_id=ZDQMVJGYPL4R7H2MSYDPVGLKN4B7FG3SCGVZMDU5MFLVISQ3DC6Q, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.347443699 (10h29m44.306540953s) [noise=true , address=104.36.82.58:4296 , piece_id=AGVDDRLZXAAXATA72TY76JNH7P6J6RSFRZFCI7FNT6SSEBNQ6HAA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.347752140 (10h29m44.30684939s) [noise=true , address=104.36.82.60:4359 , piece_id=2DZPFT7KW2BGOQZXKLQM64FS4Q7QJQPQUNTDMRPVKV5JU3KQN4MQ, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.348045599 (10h29m44.307142858s) [noise=true , address=104.36.82.38:4720 , piece_id=6BMWQEPTWJXMPD5VYOCHEERBK374TUXUNUFEH4W427CGZU33Z4CA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.348372798 (10h29m44.307470046s) [noise=true , address=104.36.82.21:4496 , piece_id=OPJLKLLJ7SUYZ4BZ325AANRWXDQJM6JPBSC6JHT3WTI5SGVRHQNA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.348673074 (10h29m44.30777033s) [noise=true , address=64.49.13.140:4511 , piece_id=T7H2CXRTLLZOTJTPVD6BYSVSPWEGB4Y3C2VCIZMZGX5F3RKRYNLA, seg_pos={0 0}, upload=10871]: Uploading piece...
Aug 9 03:55:22.911067156 (10h29m44.870164401s) [upload=10871]: Next returned nil segment
Aug 9 03:55:22.911084172 (10h29m44.870181416s) [upload=10871]: Waiting for error group managing segments...
Aug 9 03:55:23.048261951 (10h29m45.007359196s) [noise=true , address=45.91.116.14:4200 , piece_id=3NK2RKJPXS4UAJXY55I2LHQRX7D3ZPKWFG3SMSV4EZ7MHUROSGRA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.064939046 (10h29m45.024036292s) [noise=true , address=104.36.82.60:4359 , piece_id=2DZPFT7KW2BGOQZXKLQM64FS4Q7QJQPQUNTDMRPVKV5JU3KQN4MQ, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.066143838 (10h29m45.025241084s) [noise=true , address=104.36.82.47:4533 , piece_id=SR7G6S2Z3AGRWND572QYBPMKDNRK3MD6QRUMBDYLRM2ZYVNFVUTQ, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.068805794 (10h29m45.027903038s) [noise=true , address=104.36.82.59:4323 , piece_id=YVKZVOU4QTUNNHC3W3UUF3KISTPJ3HMU5QIGGFOPFRLS2R3XEF3Q, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.074735109 (10h29m45.03383234s) [noise=true , address=104.36.82.45:4913 , piece_id=A53X5FREON2B3TPU4OJA6GY7743MG63MPW6KJEQ4ICMJEV5QZMYQ, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.076585738 (10h29m45.035682976s) [noise=true , address=104.36.82.15:4267 , piece_id=6QVBPNZGKNZEH35LRTWXLMZIVNYSUXFLGF4UXA7VOBUUI4LHRSLQ, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.078307083 (10h29m45.037404316s) [noise=true , address=104.36.82.44:4872 , piece_id=ZDQMVJGYPL4R7H2MSYDPVGLKN4B7FG3SCGVZMDU5MFLVISQ3DC6Q, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.080903255 (10h29m45.040000501s) [noise=true , address=104.36.82.58:4296 , piece_id=AGVDDRLZXAAXATA72TY76JNH7P6J6RSFRZFCI7FNT6SSEBNQ6HAA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.084965825 (10h29m45.044063057s) [noise=true , address=104.36.82.2:4759 , piece_id=B35NJJJ7E6ULFD5OGVDMH4OIC77IGELIRP7AUAJWOYJAGKPDNWRA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.086897888 (10h29m45.045995124s) [noise=true , address=104.36.82.45:4907 , piece_id=BP2W2L76MWDS6W2PQRPK2P3COOD75G75D37ZUGFCY2NOWXBSZGEQ, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.088234355 (10h29m45.047331591s) [noise=true , address=104.36.82.46:4482 , piece_id=MJM7TS3NQTQJ2D4SMYDGDS2FX3AUICTMXSYJO2LE4F3F74CFBOAQ, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.090326389 (10h29m45.049423623s) [noise=true , address=104.36.82.12:4158 , piece_id=NDGI2BLPE4IMV2JWEZ5AXPHRBT6MSME4UNNGYDOKDOIA6PYSS3TA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.096118399 (10h29m45.055215633s) [noise=true , address=104.36.82.23:4458 , piece_id=5R5O366APTWEUDTZG5GWTSDBNX5S53VZPO2K2U3ZD45OEXBSPHUA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.096577595 (10h29m45.05567483s) [noise=true , address=104.36.82.38:4720 , piece_id=6BMWQEPTWJXMPD5VYOCHEERBK374TUXUNUFEH4W427CGZU33Z4CA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: <nil>
Aug 9 03:55:23.096630675 (10h29m45.05572792s) [seg_pos={0 0}, upload=10871]: Segment reached optimal threshold of 14 pieces.
Aug 9 03:55:23.096843965 (10h29m45.055941203s) [noise=true , address=104.36.82.21:4496 , piece_id=OPJLKLLJ7SUYZ4BZ325AANRWXDQJM6JPBSC6JHT3WTI5SGVRHQNA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uPsj1cL28HYcNZYjinFAwBHx4bWeSz1E1ndTDYE3y6DwbZg7w): protocol: expected piece hash; context canceled; EOF
Aug 9 03:55:23.097165055 (10h29m45.056262294s) [noise=true , address=104.36.82.16:4319 , piece_id=JZKTOUWZW7SWGC3737OGKA5R2V64ANU5ZIXNMLDA4GKCG77Y55HA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hQTG2DzecaoN1Na2mY3JJqNj2qxtJDQkhMYmRmd9hve8D7T9C): protocol: expected piece hash; context canceled; EOF
Aug 9 03:55:23.097248518 (10h29m45.056345754s) [noise=true , address=64.49.13.140:4511 , piece_id=T7H2CXRTLLZOTJTPVD6BYSVSPWEGB4Y3C2VCIZMZGX5F3RKRYNLA, seg_pos={0 0}, upload=10871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1F79nPRFzy9hjCq5KYvLy6bZERw9cEvczCN2o66Cf6ccKEpkAk): protocol: write tcp 139.59.144.22:59802->64.49.13.140:4511: use of closed network connection; write tcp 139.59.144.22:59802->64.49.13.140:4511: use of closed network connection; piecestore: piecestore close: write tcp 139.59.144.22:59802->64.49.13.140:4511: use of closed network connection
Aug 9 03:55:23.097330100 (10h29m45.056427345s) [noise=true , address=104.36.82.39:4739 , piece_id=R3OAUKXTNBEEZMXX3CJDT2CASE3HLBXNPKD4EGRRFFFF7EGTQ2YQ, seg_pos={0 0}, upload=10871]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A4nWQsS43tSv3atCnj6CB2ZGKFxQUHJp8a7ftP9y1XumeFk33): protocol: expected piece hash; context canceled; EOF
Aug 9 03:55:23.097345526 (10h29m45.056442772s) [seg_pos={0 0}, upload=10871]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Aug 9 03:55:23.097442993 (10h29m45.056540238s) [upload=10871]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Aug 9 03:55:23.097448165 (10h29m45.056545413s) [upload=10871]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Aug 9 03:55:23.258254104 (10h29m45.21735135s) [upload=10871]: Done uploading segments...
Aug 9 03:55:23.258294964 (10h29m45.217392209s) [upload=10871]: Upload finished. err: <nil>
Duration: 1.598835749s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
6 out of 60 is failed (threshold=3) [.....................................X..XXX.....X.X.........] duration:
303.654002ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-09 03:55:13 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 303.725364ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [.........................X...................X..........X...] duration:
1.123501826s
FAIL (2025-08-09 03:55:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.123531707s
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
10.087088991s
FAIL (2025-08-09 03:55:06 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 10.087119066s
edge
authservice
http
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [........................X............X......................] duration:
979.937111ms
FAIL (2025-08-09 03:54:56 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:38:21.391404361 (10h12m43.350501607s) [upload=10580]: Starting upload Aug 9 03:38:21.391499574 (10h12m43.350596824s) [upload=10580]: Done starting upload Aug 9 03:38:21.391580214 (10h12m43.35067745s) [upload=10580]: Uploading segments... Aug 9 03:38:21.391626103 (10h12m43.350723357s) [upload=10580]: Got next segment. Inline: true Aug 9 03:38:21.391640756 (10h12m43.350738001s) [upload=10580]: Waiting for error group managing segments... Aug 9 03:38:21.391684974 (10h12m43.350782221s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:38:21.391714671 (10h12m43.350811918s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 9 03:38:21.391718801 (10h12m43.35081605s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 9 03:38:21.563255973 (10h12m43.522353225s) [upload=10580]: Done uploading segments... Aug 9 03:38:21.563291106 (10h12m43.522388342s) [upload=10580]: Upload finished. err: metaclient: Too Many Requests
Duration: 979.965797ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
6 out of 60 is failed (threshold=3) [......................................X..XXX.....X.X........] duration:
419.439129ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-09 03:54:46 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 419.462875ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [..........................X...................X..........X..] duration:
1.073402099s
FAIL (2025-08-09 03:54:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.073421839s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
2.331422287s
FAIL (2025-08-09 03:54:30 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:37:30.833561496 (10h11m52.792658745s) [upload=10565]: Starting upload Aug 9 03:37:30.833618177 (10h11m52.792715423s) [upload=10565]: Done starting upload Aug 9 03:37:30.833943401 (10h11m52.79304065s) [upload=10565]: Uploading segments... Aug 9 03:37:30.834025401 (10h11m52.793122657s) [upload=10565]: Got next segment. Inline: false Aug 9 03:37:30.834070320 (10h11m52.793167571s) [upload=10565]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:37:30.834077339 (10h11m52.793174589s) [upload=10565]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 9 03:37:30.921740356 (10h11m52.880837612s) [upload=10565]: Next segment err: metaclient: Too Many Requests Aug 9 03:37:30.921770756 (10h11m52.880868006s) [upload=10565]: Done uploading segments... Aug 9 03:37:30.921792467 (10h11m52.880889718s) [upload=10565]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.331456296s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................X....................X.........X.........] duration:
2.52976825s
FAIL (2025-08-09 03:54:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.529816637s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.........................X............X.....................] duration:
835.201688ms
FAIL (2025-08-09 03:54:17 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:38:21.391404361 (10h12m43.350501607s) [upload=10580]: Starting upload Aug 9 03:38:21.391499574 (10h12m43.350596824s) [upload=10580]: Done starting upload Aug 9 03:38:21.391580214 (10h12m43.35067745s) [upload=10580]: Uploading segments... Aug 9 03:38:21.391626103 (10h12m43.350723357s) [upload=10580]: Got next segment. Inline: true Aug 9 03:38:21.391640756 (10h12m43.350738001s) [upload=10580]: Waiting for error group managing segments... Aug 9 03:38:21.391684974 (10h12m43.350782221s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:38:21.391714671 (10h12m43.350811918s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 9 03:38:21.391718801 (10h12m43.35081605s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 9 03:38:21.563255973 (10h12m43.522353225s) [upload=10580]: Done uploading segments... Aug 9 03:38:21.563291106 (10h12m43.522388342s) [upload=10580]: Upload finished. err: metaclient: Too Many Requests
Duration: 835.233949ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
2.413488993s
FAIL (2025-08-09 03:54:14 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:37:30.833561496 (10h11m52.792658745s) [upload=10565]: Starting upload Aug 9 03:37:30.833618177 (10h11m52.792715423s) [upload=10565]: Done starting upload Aug 9 03:37:30.833943401 (10h11m52.79304065s) [upload=10565]: Uploading segments... Aug 9 03:37:30.834025401 (10h11m52.793122657s) [upload=10565]: Got next segment. Inline: false Aug 9 03:37:30.834070320 (10h11m52.793167571s) [upload=10565]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:37:30.834077339 (10h11m52.793174589s) [upload=10565]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 9 03:37:30.921740356 (10h11m52.880837612s) [upload=10565]: Next segment err: metaclient: Too Many Requests Aug 9 03:37:30.921770756 (10h11m52.880868006s) [upload=10565]: Done uploading segments... Aug 9 03:37:30.921792467 (10h11m52.880889718s) [upload=10565]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.413516497s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [...........................X...................X..........X.] duration:
1.017918991s
FAIL (2025-08-09 03:53:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1.017929933s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
3 out of 60 is failed (threshold=3) [............................X...................X..........X] duration:
30.000295432s
FAIL (2025-08-09 03:53:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000393721s
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
308.242598ms
FAIL (2025-08-09 03:53:52 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 308.269825ms
edge
authservice
http
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [..........................X............X....................] duration:
839.006407ms
FAIL (2025-08-09 03:53:45 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:38:21.391404361 (10h12m43.350501607s) [upload=10580]: Starting upload Aug 9 03:38:21.391499574 (10h12m43.350596824s) [upload=10580]: Done starting upload Aug 9 03:38:21.391580214 (10h12m43.35067745s) [upload=10580]: Uploading segments... Aug 9 03:38:21.391626103 (10h12m43.350723357s) [upload=10580]: Got next segment. Inline: true Aug 9 03:38:21.391640756 (10h12m43.350738001s) [upload=10580]: Waiting for error group managing segments... Aug 9 03:38:21.391684974 (10h12m43.350782221s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:38:21.391714671 (10h12m43.350811918s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 9 03:38:21.391718801 (10h12m43.35081605s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 9 03:38:21.563255973 (10h12m43.522353225s) [upload=10580]: Done uploading segments... Aug 9 03:38:21.563291106 (10h12m43.522388342s) [upload=10580]: Upload finished. err: metaclient: Too Many Requests
Duration: 839.037436ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
6 out of 60 is failed (threshold=3) [.......................................X..XXX.....X.X.......] duration:
321.92995ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-09 03:53:39 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 321.948422ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................X....................X.........X........] duration:
2.571381016s
FAIL (2025-08-09 03:53:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.571406974s
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
322.632368ms
FAIL (2025-08-09 03:53:14 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 322.667192ms
edge
authservice
http
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
2.824144929s
FAIL (2025-08-09 03:52:59 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:37:30.833561496 (10h11m52.792658745s) [upload=10565]: Starting upload Aug 9 03:37:30.833618177 (10h11m52.792715423s) [upload=10565]: Done starting upload Aug 9 03:37:30.833943401 (10h11m52.79304065s) [upload=10565]: Uploading segments... Aug 9 03:37:30.834025401 (10h11m52.793122657s) [upload=10565]: Got next segment. Inline: false Aug 9 03:37:30.834070320 (10h11m52.793167571s) [upload=10565]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:37:30.834077339 (10h11m52.793174589s) [upload=10565]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 9 03:37:30.921740356 (10h11m52.880837612s) [upload=10565]: Next segment err: metaclient: Too Many Requests Aug 9 03:37:30.921770756 (10h11m52.880868006s) [upload=10565]: Done uploading segments... Aug 9 03:37:30.921792467 (10h11m52.880889718s) [upload=10565]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.824174964s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
6 out of 60 is failed (threshold=3) [........................................X..XXX.....X.X......] duration:
312.914621ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-09 03:52:48 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 312.937535ms
edge
gateway
vhost
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [...........................X............X...................] duration:
839.205238ms
FAIL (2025-08-09 03:52:46 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:38:21.391404361 (10h12m43.350501607s) [upload=10580]: Starting upload Aug 9 03:38:21.391499574 (10h12m43.350596824s) [upload=10580]: Done starting upload Aug 9 03:38:21.391580214 (10h12m43.35067745s) [upload=10580]: Uploading segments... Aug 9 03:38:21.391626103 (10h12m43.350723357s) [upload=10580]: Got next segment. Inline: true Aug 9 03:38:21.391640756 (10h12m43.350738001s) [upload=10580]: Waiting for error group managing segments... Aug 9 03:38:21.391684974 (10h12m43.350782221s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:38:21.391714671 (10h12m43.350811918s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 9 03:38:21.391718801 (10h12m43.35081605s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 9 03:38:21.563255973 (10h12m43.522353225s) [upload=10580]: Done uploading segments... Aug 9 03:38:21.563291106 (10h12m43.522388342s) [upload=10580]: Upload finished. err: metaclient: Too Many Requests
Duration: 839.247445ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................X....................X.........X.......] duration:
2.63570053s
FAIL (2025-08-09 03:52:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.635723337s
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
5.298981361s
FAIL (2025-08-09 03:52:30 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 5.299008906s
edge
authservice
http
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.............................X...................X..........] duration:
1.002355133s
FAIL (2025-08-09 03:52:24 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 1.002372581s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [............................X............X..................] duration:
818.468655ms
FAIL (2025-08-09 03:51:58 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:38:21.391404361 (10h12m43.350501607s) [upload=10580]: Starting upload Aug 9 03:38:21.391499574 (10h12m43.350596824s) [upload=10580]: Done starting upload Aug 9 03:38:21.391580214 (10h12m43.35067745s) [upload=10580]: Uploading segments... Aug 9 03:38:21.391626103 (10h12m43.350723357s) [upload=10580]: Got next segment. Inline: true Aug 9 03:38:21.391640756 (10h12m43.350738001s) [upload=10580]: Waiting for error group managing segments... Aug 9 03:38:21.391684974 (10h12m43.350782221s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:38:21.391714671 (10h12m43.350811918s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 9 03:38:21.391718801 (10h12m43.35081605s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 9 03:38:21.563255973 (10h12m43.522353225s) [upload=10580]: Done uploading segments... Aug 9 03:38:21.563291106 (10h12m43.522388342s) [upload=10580]: Upload finished. err: metaclient: Too Many Requests
Duration: 818.494202ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................X....................X.........X......] duration:
2.513708289s
FAIL (2025-08-09 03:51:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.513737006s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
2.432267763s
FAIL (2025-08-09 03:51:53 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:37:30.833561496 (10h11m52.792658745s) [upload=10565]: Starting upload Aug 9 03:37:30.833618177 (10h11m52.792715423s) [upload=10565]: Done starting upload Aug 9 03:37:30.833943401 (10h11m52.79304065s) [upload=10565]: Uploading segments... Aug 9 03:37:30.834025401 (10h11m52.793122657s) [upload=10565]: Got next segment. Inline: false Aug 9 03:37:30.834070320 (10h11m52.793167571s) [upload=10565]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:37:30.834077339 (10h11m52.793174589s) [upload=10565]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 9 03:37:30.921740356 (10h11m52.880837612s) [upload=10565]: Next segment err: metaclient: Too Many Requests Aug 9 03:37:30.921770756 (10h11m52.880868006s) [upload=10565]: Done uploading segments... Aug 9 03:37:30.921792467 (10h11m52.880889718s) [upload=10565]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.432302095s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
6 out of 60 is failed (threshold=3) [.........................................X..XXX.....X.X.....] duration:
312.382132ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-09 03:51:42 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 312.408682ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [..............................X...................X.........] duration:
1.077398065s
FAIL (2025-08-09 03:51:39 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 1.077469292s
edge
gateway
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
325.949721ms
FAIL (2025-08-09 03:51:28 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 325.998021ms
edge
authservice
http
us-select-1
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
6 out of 60 is failed (threshold=3) [..........................................X..XXX.....X.X....] duration:
325.519338ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-09 03:51:23 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 325.542024ms
edge
gateway
vhost
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................X....................X.........X.....] duration:
2.663288194s
FAIL (2025-08-09 03:51:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.663324328s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
2.448015769s
FAIL (2025-08-09 03:50:52 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:37:30.833561496 (10h11m52.792658745s) [upload=10565]: Starting upload Aug 9 03:37:30.833618177 (10h11m52.792715423s) [upload=10565]: Done starting upload Aug 9 03:37:30.833943401 (10h11m52.79304065s) [upload=10565]: Uploading segments... Aug 9 03:37:30.834025401 (10h11m52.793122657s) [upload=10565]: Got next segment. Inline: false Aug 9 03:37:30.834070320 (10h11m52.793167571s) [upload=10565]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:37:30.834077339 (10h11m52.793174589s) [upload=10565]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Aug 9 03:37:30.921740356 (10h11m52.880837612s) [upload=10565]: Next segment err: metaclient: Too Many Requests Aug 9 03:37:30.921770756 (10h11m52.880868006s) [upload=10565]: Done uploading segments... Aug 9 03:37:30.921792467 (10h11m52.880889718s) [upload=10565]: Upload finished. err: metaclient: Too Many Requests
Duration: 2.448044066s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.............................X............X.................] duration:
968.07111ms
FAIL (2025-08-09 03:50:50 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:38:21.391404361 (10h12m43.350501607s) [upload=10580]: Starting upload Aug 9 03:38:21.391499574 (10h12m43.350596824s) [upload=10580]: Done starting upload Aug 9 03:38:21.391580214 (10h12m43.35067745s) [upload=10580]: Uploading segments... Aug 9 03:38:21.391626103 (10h12m43.350723357s) [upload=10580]: Got next segment. Inline: true Aug 9 03:38:21.391640756 (10h12m43.350738001s) [upload=10580]: Waiting for error group managing segments... Aug 9 03:38:21.391684974 (10h12m43.350782221s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:38:21.391714671 (10h12m43.350811918s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 9 03:38:21.391718801 (10h12m43.35081605s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 9 03:38:21.563255973 (10h12m43.522353225s) [upload=10580]: Done uploading segments... Aug 9 03:38:21.563291106 (10h12m43.522388342s) [upload=10580]: Upload finished. err: metaclient: Too Many Requests
Duration: 968.110851ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)antiblip:
6 out of 60 is failed (threshold=3) [...........................................X..XXX.....X.X...] duration:
365.47274ms remote:
"tcp:45.38.24.2:443" target:
https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410
FAIL (2025-08-09 03:50:41 UTC)
orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded
Duration: 365.497266ms
edge
gateway
vhost
us-select-1
|
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
309.885912ms
FAIL (2025-08-09 03:50:33 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": context deadline exceeded
Duration: 309.924793ms
edge
authservice
http
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [..............................X............X................] duration:
1.064845776s
FAIL (2025-08-09 03:50:30 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 9 03:38:21.391404361 (10h12m43.350501607s) [upload=10580]: Starting upload Aug 9 03:38:21.391499574 (10h12m43.350596824s) [upload=10580]: Done starting upload Aug 9 03:38:21.391580214 (10h12m43.35067745s) [upload=10580]: Uploading segments... Aug 9 03:38:21.391626103 (10h12m43.350723357s) [upload=10580]: Got next segment. Inline: true Aug 9 03:38:21.391640756 (10h12m43.350738001s) [upload=10580]: Waiting for error group managing segments... Aug 9 03:38:21.391684974 (10h12m43.350782221s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 9 03:38:21.391714671 (10h12m43.350811918s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 9 03:38:21.391718801 (10h12m43.35081605s) [upload=10580]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 9 03:38:21.563255973 (10h12m43.522353225s) [upload=10580]: Done uploading segments... Aug 9 03:38:21.563291106 (10h12m43.522388342s) [upload=10580]: Upload finished. err: metaclient: Too Many Requests
Duration: 1.06487377s
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_eu_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [...............................X...................X........] duration:
2.43765626s
FAIL (2025-08-09 03:50:30 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 2.437681299s
edge
gateway
us-select-1
|