kube-dns and kubernetes-dashboard pods status are CrashLoopBackOff -


i setup multi-node kubernetes cluster (3 etcds, 2 masters , 2 nodes) in openstack following https://coreos.com/kubernetes/docs/latest/getting-started.html

all vm has coreos 1185.3.0

kubectl version client version: version.info{major:"1", minor:"3", gitversion:"v1.4.3", gitcommit:"ae4550cc9c89a593bcda6678df201db1b208133b", gittreestate:"clean", builddate:"2016-08-26t18:13:23z", goversion:"go1.6.2", compiler:"gc", platform:"linux/amd64"} server version: version.info{major:"1", minor:"4", gitversion:"v1.4.0+coreos.0", gitcommit:"278a1f7034bdba61cba443722647da1a8204a6fc", gittreestate:"clean", builddate:"2016-09-26t20:48:37z", goversion:"go1.6.3", compiler:"gc", platform:"linux/amd64"}

kubectl nodes return cluster healthy

name            status                     age 172.29.0.157    ready,schedulingdisabled   1d 172.29.0.158    ready,schedulingdisabled   1d 172.24.0.120    ready                      1d 172.24.0.121    ready                      1d

kubectl pods --namespace=kube-system return kube-dns , kubernetes-dashboard pods status crashloopbackoff

name                                   ready     status             restarts   age heapster-v1.2.0-3646253287-xweg5       2/2       running            0          2h kube-apiserver-172.29.0.157            1/1       running            2          1d kube-apiserver-172.29.0.158            1/1       running            1          1d kube-controller-manager-172.29.0.157   1/1       running            2          1d kube-controller-manager-172.29.0.158   1/1       running            1          1d kube-dns-v19-h7qyh                     2/3       crashloopbackoff   13         2h kube-proxy-172.24.0.120                1/1       running            2          36m kube-proxy-172.24.0.121                1/1       running            2          37m kube-proxy-172.29.0.157                1/1       running            2          1d kube-proxy-172.29.0.158                1/1       running            1          1d kube-scheduler-172.29.0.157            1/1       running            2          1d kube-scheduler-172.29.0.158            1/1       running            1          1d kubernetes-dashboard-v1.4.0-t2lpu      0/1       crashloopbackoff   12         2h

can tell me know how figure out exact issue here?

update:

i able logs of kube-dns , kubernetes-dashboard containers. seems certificate issue when trying call kubernetes api. have recreated certificate , replace them.

setting master , worker instructions, https://coreos.com/kubernetes/docs/latest/deploy-master.html https://coreos.com/kubernetes/docs/latest/deploy-workers.html

masters fronted load balancer.

finally restarted kubernetes 2 master vms , 2 node vms. problem still persists in kube-dns , kubernetes-dashboard.

kube-dns container logs

docker logs c8c82e68cde9 i1111 16:28:25.097452       1 server.go:94] using https://10.3.0.1:443 kubernetes master, kubernetes api: <nil> i1111 16:28:25.103598       1 server.go:99] v1.4.0-alpha.2.1652+c69e3d32a29cfa-dirty i1111 16:28:25.103789       1 server.go:101] flag: --alsologtostderr="false" i1111 16:28:25.103928       1 server.go:101] flag: --dns-port="10053" i1111 16:28:25.104185       1 server.go:101] flag: --domain="cluster.local." i1111 16:28:25.104301       1 server.go:101] flag: --federations="" i1111 16:28:25.104465       1 server.go:101] flag: --healthz-port="8081" i1111 16:28:25.104607       1 server.go:101] flag: --kube-master-url="" i1111 16:28:25.104718       1 server.go:101] flag: --kubecfg-file="" i1111 16:28:25.104831       1 server.go:101] flag: --log-backtrace-at=":0" i1111 16:28:25.104945       1 server.go:101] flag: --log-dir="" i1111 16:28:25.105056       1 server.go:101] flag: --log-flush-frequency="5s" i1111 16:28:25.105188       1 server.go:101] flag: --logtostderr="true" i1111 16:28:25.105302       1 server.go:101] flag: --stderrthreshold="2" i1111 16:28:25.105412       1 server.go:101] flag: --v="0" i1111 16:28:25.105520       1 server.go:101] flag: --version="false" i1111 16:28:25.105632       1 server.go:101] flag: --vmodule="" i1111 16:28:25.105853       1 server.go:138] starting skydns server. listening on port:10053 i1111 16:28:25.106185       1 server.go:145] skydns: metrics enabled on : /metrics: i1111 16:28:25.106367       1 dns.go:167] waiting service: default/kubernetes i1111 16:28:25.108281       1 logs.go:41] skydns: ready queries on cluster.local. tcp://0.0.0.0:10053 [rcache 0] i1111 16:28:25.108469       1 logs.go:41] skydns: ready queries on cluster.local. udp://0.0.0.0:10053 [rcache 0] e1111 16:28:25.176270       1 reflector.go:214] pkg/dns/dns.go:155: failed list *api.endpoints: server has asked client provide credentials (get endpoints) i1111 16:28:25.176774       1 dns.go:173] ignoring error while waiting service default/kubernetes: server has asked client provide credentials (get services kubernetes). sleeping 1s before retrying.

kubernetes-dashboard container logs

docker logs b1d3b0fa617a starting http server on port 9090 creating api server client https://10.3.0.1:443 error while initializing connection kubernetes apiserver. means cluster misconfigured (e.g., has invalid apiserver certificates or service accounts configuration) or --apiserver-host param points server not exist. reason: server has asked client provide credentials

kubernetes node logs

journalctl -u kubelet -f failed list *api.node: https://{load_balancer_ip}/api/v1/nodes?fieldselector=metadata.name%3d172.24.0.121&resourceversion=0: x509: certificate signed unknown authority (possibly because of "crypto/rsa: verification error" while trying verify candidate authority certificate "kube-ca")

i followed https://coreos.com/kubernetes/docs/latest/openssl.html when generating certs.

api server certs generated below openssl config

[req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicconstraints = ca:false keyusage = nonrepudiation, digitalsignature, keyencipherment subjectaltname = @alt_names [alt_names] dns.1 = kubernetes dns.2 = kubernetes.default dns.3 = kubernetes.default.svc dns.4 = kubernetes.default.svc.cluster.local ip.1 = ${k8s_service_ip} ip.2 = ${load_balancer_ip}

am missing here ?

thanks


Comments

Post a Comment

Popular posts from this blog

sql server - Cannot query correctly (MSSQL - PHP - JSON) -

i have little problem on project. , turns out query result returns nothing when value of first name or last name has 'ñ' in it. here's code config.php: <?php ini_set('mssql.charset', 'utf-8'); header('content-type: text/html; charset=utf-8'); $servername = "192.168.1.21"; /* ip add of db server */ $connection = array("database" => "db", "uid" => "?", "pwd" => "***"); $conn = sqlsrv_connect($servername, $connection); if(!$conn){ echo "connection not established."; die(print_r(sqlsrv_errors(), true)); } ?> myquery: $idnumber = $_post["idnum"]; $response = array(); $query2 = "select clname, cfname, cmname, cqualifier student cidno = '$idnumber'"; try{ $stmt2 = sqlsrv_query($conn, $query2); } catch(pdoexeption $ex){ $response["success"] = 0; $response["message"] = "d...
Read more

php - trouble displaying mysqli database results in correct order -

i have table in database stores events , date expire. want display events on website display current events first expired events after? i have tried following sql $sql = "select noticeid, notice, noticedesc, noticeimg, noticedate, expirydate tbl_notices group noticeid ,expired order expirydate asc"; but returns expired results first due expirydate being ordered asc in query. my tbl_notice structure looks this noticeid => int(4) notice => varchar(100) noticedesc => text noticedate => timestamp noticeimg => varchar(40) expirydate => datetime expired => int(1) 0 current or 1 expired my php code is $sql = "select noticeid, notice, noticedesc, noticeimg, noticedate, expirydate tbl_notices group noticeid ,expired order expirydate asc"; $result = mysqli_query($conn,$sql); if ($result->num_rows > 0) { // output data of each row while($row = $result->fetch_assoc()) { ...
Read more

C++ Linked List -

i trying create linked list download txt file , use linked list handle file line line. when handling downloaded linked list operations performed on such text editor would. encountering problems however. seems "node(string value)" section of code has wrong though original node() declaration no arguments passes. unable figure out quite is. node.h class node { public: node(); node(string value); void setnext(node *nextnode); // allows user set "next" pointer of node points friend class linkedlist; private: string data; // data box node* next; // pointer box }; node.cpp # include <string> # include "node.h" using namespace std; node::node() { data = ""; next = null; } node::node(string value) { data = value; next = null; } void node::setnext(node *nextnode) // allows user set "next" pointer of node points { this->next = nextnode; } your #include <string> ...
Read more