IPv6 Neighbour table overflow

in Network

這兩天一直在 syslog 上看到 Neighbour table overflow

/var/log/syslog
1
2
Apr 30 06:27:03 host-42 kernel: [72924.290265] net_ratelimit: 1739 callbacks suppressed
Apr 30 06:27:03 host-42 kernel: [72924.290269] IPv6: Neighbour table overflow

每兩分鐘就出現6~7筆,數量非常的多
如果不處理他，放任它繼續增長，過一兩天後系統就會 kernel panic。

Continue reading

OpenStack Glance Image-list 401

in OpenStack

架設 OpenStack Juno ，安裝完 Glance 後進行測試時，glance 指令會一直出現 401 錯誤。

1
2
3
4
root@localhost:~# glance --debug image-list
curl -i -X GET -H 'User-Agent: python-glanceclient' -H 'Content-Type: application/octet-stream' -H 'Accept-Encoding: gzip, deflate, compress' -H 'Accept: */*' -H 'X-Auth-Token: ***' http://140.114.91.220:9292/v1/images/detail?sort_key=name&sort_dir=asc&limit=20
Request returned failure status 401.
Invalid OpenStack Identity credentials.
Continue reading

Install Rally 筆記 (OpenStack benchmark)

in OpenStack

Rally 是一套 OpenStack benchmark tool。關於 Rally 的介紹本文就不再贅述。本篇安裝時是以 OpenStack 官方 Wiki 的教學為參考內容，紀錄安裝與使用 Rally 的筆記。

實驗環境

Item Value
Operating System Ubuntu 12.04 x64
Python version Python 2.7
OpenStack version Grizzly (2013)

Install Rally

一開始我是想要安裝在 OpenStack controller node 上，但是一直裝不起來，看起來似乎是 python module 的版本問題，不過我沒有仔細研究到底是哪個 module 的哪個版本卡住。

直接在 OpenStack 上開了一個新的 VM，一樣是選擇 Ubuntu 12.04，就可以直接安裝。(不過現在回想起來，應該要用 virtualenv 來裝就可以解決，也不會影響原本的系統。)

Continue reading

Ceph pgs stuck in 'incomplete' state ,ops blocked

in Storage

Ceph OSD 又再次發生 disk failure，結果在手動修復硬碟時操作不當，整個 disk partition table 都消失了
即使把備份的 disk patition table 寫回去之後，依然無法解決問題。

無奈之下，硬是將 Ceph cluster 開啟 (少一個 OSD)

執行 ceph health detail 得到以下狀態:

1
2
3
4
5
6
pg 0.1f is stuck inactive since forever, current state incomplete, last acting [3]
pg 0.1f is stuck unclean since forever, current state incomplete, last acting [3]
pg 0.1f is incomplete, acting [3]
32 ops are blocked > 32.768 sec on osd.3
32 ops are blocked > 32.768 sec on osd.3
1 osds have slow requests
Continue reading

OpenStack Nova cannot reboot instance - Unable to create tap device

in OpenStack

早上因為某種因素，將 nova compute 強制重開機。當開機完成之後，使用 nova reboot --hard <server> 的方式，想要開啟instance 卻失敗，在 /var/log/nova/nova-compute.log 中看到以下錯誤訊息:

1
2015-02-11 16:10:54.110 ERROR nova.compute.manager [req-a3d9cf35-82ee-4857-b69d-99ef0c8ca753 b6a90e8c63ad4612917655fb9b04ad92 ecb687200c6a4574bdaf3ea3633c6b3f] [instance: 7bdad622-dd70-49d7-89ca-827d2e86367f] Cannot reboot instance: Unable to create tap device tape10b9639-d8: Device or resource busy
Continue reading

OpenStack Nova error: libVirt cannot get CPU affinity of process 30619

in OpenStack

Nova-compute 開不起來，在 log 中發現:

1
2
3
4
5
6
7
8
9
10
2015-02-11 15:34:05.511 30827 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
2015-02-11 15:34:05.511 30827 TRACE nova     result = proxy_call(self._autowrap, f, *args, **kwargs)
2015-02-11 15:34:05.511 30827 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call
2015-02-11 15:34:05.511 30827 TRACE nova     rv = execute(f,*args,**kwargs)
2015-02-11 15:34:05.511 30827 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
2015-02-11 15:34:05.511 30827 TRACE nova     rv = meth(*args,**kwargs)
2015-02-11 15:34:05.511 30827 TRACE nova   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 2096, in vcpus
2015-02-11 15:34:05.511 30827 TRACE nova     if ret == -1: raise libvirtError ('virDomainGetVcpus() failed', dom=self)
2015-02-11 15:34:05.511 30827 TRACE nova libvirtError: cannot get CPU affinity of process 30619: No such process
2015-02-11 15:34:05.511 30827 TRACE nova
Continue reading

古振浩 Chen-Hao Ku

Taiwanese working in ShangHai


Backend Develop Engineer, UCloud


ShangHai, China