PengYao's Webloghttps://pengyao.org/2015-01-27T00:00:00+08:00Salt zmq_filtering测试2015-01-27T00:00:00+08:002015-01-27T00:00:00+08:00pengyaotag:pengyao.org,2015-01-27:/salt-zmq-filtering.html<p class="first last">Salt 2014.7新增了zmq_filtering配置项, 利用zeromq PUB/SUB Envelopes技术, 可以实现消息只发送到target minion(目前只支持list target)</p>
<p>Salt 2014.7新增 <a class="reference external" href="https://github.com/saltstack/salt/pull/13285">zmq_filtering</a> 配置项, 基于zeromq PUB-SUB <a class="reference external" href="http://zguide.zeromq.org/page:all#Pub-Sub-Message-Envelopes">Envelopes</a> 技术,
在Master端(publisher)进行message过滤(ZeroMQ 3.0+版本,之前版本是在subscriber端进行过滤), 以实现如果该指令只是少量主机执行的话,
只将指令发送到匹配的Minion端, 而并非发送到所有的Minion端. 需要注意的是, 目前zmq_filtering只作用list target, 即使用-L来指定target.
本文将对其进行功能及效果测试.</p>
<div class="section" id="section-1">
<h2>前置阅读</h2>
<ul class="simple">
<li>ZeroMQ PUB-SUB Message Envelopes: <a class="reference external" href="http://zguide.zeromq.org/page:all#Pub-Sub-Message-Envelopes">http://zguide.zeromq.org/page:all#Pub-Sub-Message-Envelopes</a></li>
</ul>
</div>
<div class="section" id="section-2">
<h2>环境说明</h2>
<ul class="simple">
<li>OS: CentOS 6.5 X86_64</li>
<li>Salt: Master/Minion架构, 2014.7.1版本</li>
</ul>
</div>
<div class="section" id="section-3">
<h2>开工</h2>
<div class="section" id="zmq-filtering">
<h3>配置zmq_filtering</h3>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">zmq_filtering参数在Master端及Minion端均需要配置才能生效, 默认为False</p>
</div>
<p>Master端配置zmq_filtering</p>
<div class="highlight"><pre><span></span><span class="nb">echo</span><span class="w"> </span><span class="s2">"zmq_filtering: True"</span><span class="w"> </span>>><span class="w"> </span>/etc/salt/master
service<span class="w"> </span>salt-master<span class="w"> </span>restart
</pre></div>
<p>Minion端配置zmq_filtering</p>
<div class="highlight"><pre><span></span><span class="nb">echo</span><span class="w"> </span><span class="s2">"zmq_filtering: True"</span><span class="w"> </span>>><span class="w"> </span>/etc/salt/minion
service<span class="w"> </span>salt-minion<span class="w"> </span>restart
</pre></div>
<p>操作完毕后, Master进行test.ping测试</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>test.ping
</pre></div>
<p>输出如下内容:</p>
<pre class="literal-block">
minion-02.example.com:
True
minion-01.example.com:
True
</pre>
</div>
<div class="section" id="zmq-filtering-1">
<h3>测试zmq_filtering</h3>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">以下操作如非特别声明, 均在Master端进行</p>
</div>
<p>在Master端开启一个新的控制台, 使用tcpdump进行抓包</p>
<div class="highlight"><pre><span></span>tcpdump<span class="w"> </span>-i<span class="w"> </span>eth0<span class="w"> </span>src<span class="w"> </span>port<span class="w"> </span><span class="m">4505</span>
</pre></div>
<p>使用Globbing target进行测试</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'minion-01.example.com'</span><span class="w"> </span>test.ping
</pre></div>
<p>tcpdump抓包结果如下:</p>
<pre class="literal-block">
07:21:18.104954 IP salt-master.example.com.4505 > salt-minion-01.example.com.58024: Flags [P.], seq 396:602, ack 1, win 227, options [nop,nop,TS val 4253627 ecr 4239803], length 206
07:21:18.105190 IP salt-master.example.com.4505 > salt-minion-02.example.com.33595: Flags [P.], seq 396:602, ack 1, win 227, options [nop,nop,TS val 4253627 ecr 4294907239], length 206
</pre>
<p>从结果来看, 虽然指定了minion-01.example.com, 因为Salt是PUB-SUB结构, 消息均会发送到所有的Minion</p>
<p>使用List target进行测试</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span>-L<span class="w"> </span><span class="s1">'minion-01.example.com'</span><span class="w"> </span>test.ping
</pre></div>
<p>tcpdump抓包结果如下:</p>
<pre class="literal-block">
07:23:35.378587 IP salt-master.example.com.4505 > salt-minion-01.example.com.58024: Flags [P.], seq 602:839, ack 1, win 227, options [nop,nop,TS val 4390900 ecr 4245316], length 237
</pre>
<p>从结果看, 在Master进行了过滤, 虽然是PUB-SUB, 但消息只发送给了salt-minion-01.example.com, 并没有发送到其他Minion上, 达到了zmq_filtering的效果.</p>
</div>
<div class="section" id="zeromq-pub-sub-message-envelopes">
<h3>测试ZeroMQ PUB-SUB Message Envelopes性能</h3>
<p>开启zmq_filtering, 如果不是所有Minion均需要执行的操作, 通过在Master端进行消息过滤, 能够大大降低Master端发送指令时的带宽消耗, 那么zmq_filtering的性能又如何?</p>
<p>由于zmq_filtering只是利用ZeroMQ的PUB-SUB Message Envelopes, 其性能测试个人觉得只需要测试ZeroMQ PUB/SUB即可. 因此就假设了如下极端场景:</p>
<ul class="simple">
<li>publisher 1节点, subscriber 1000节点(单节点开启1000个线程)</li>
<li>进行1000次消息发送, 每条消息均需要发送到所有subscriber</li>
</ul>
<p>直接上代码:</p>
<p><strong>publisher.py</strong></p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">zmq</span>
<span class="kn">import</span> <span class="nn">time</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="kn">import</span> <span class="nn">hashlib</span>
<span class="k">def</span> <span class="nf">pub</span><span class="p">():</span>
<span class="n">context</span> <span class="o">=</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">()</span>
<span class="n">socket</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">PUB</span><span class="p">)</span>
<span class="n">socket</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="s2">"tcp://*:5556"</span><span class="p">)</span>
<span class="k">return</span> <span class="n">context</span><span class="p">,</span> <span class="n">socket</span>
<span class="k">def</span> <span class="nf">sub_ids</span><span class="p">(</span><span class="n">subs</span><span class="o">=</span><span class="mi">100</span><span class="p">):</span>
<span class="n">sub_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">each_sub</span> <span class="ow">in</span> <span class="n">xrange</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">subs</span> <span class="o">+</span> <span class="mi">1</span><span class="p">):</span>
<span class="n">idx</span> <span class="o">=</span> <span class="n">hashlib</span><span class="o">.</span><span class="n">md5</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">each_sub</span><span class="p">))</span><span class="o">.</span><span class="n">hexdigest</span><span class="p">()</span>
<span class="n">sub_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">idx</span><span class="p">)</span>
<span class="k">return</span> <span class="n">sub_list</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">broadcast</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">times</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">subs</span><span class="o">=</span><span class="mi">100</span><span class="p">):</span>
<span class="n">context</span><span class="p">,</span> <span class="n">socket</span> <span class="o">=</span> <span class="n">pub</span><span class="p">()</span>
<span class="n">message</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'tgt_type'</span><span class="p">:</span> <span class="s1">'glob'</span><span class="p">,</span> <span class="s1">'jid'</span><span class="p">:</span> <span class="s1">'20150106053023956920'</span><span class="p">,</span> <span class="s1">'tgt'</span><span class="p">:</span> <span class="s1">'*'</span><span class="p">,</span> <span class="s1">'ret'</span><span class="p">:</span> <span class="s1">''</span><span class="p">,</span> <span class="s1">'user'</span><span class="p">:</span> <span class="s1">'sudo_vagrant'</span><span class="p">,</span> <span class="s1">'arg'</span><span class="p">:</span> <span class="p">[],</span> <span class="s1">'fun'</span><span class="p">:</span> <span class="s1">'test.ping'</span><span class="p">}</span>
<span class="n">message</span> <span class="o">=</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">broadcast</span><span class="p">:</span>
<span class="n">sub_list</span> <span class="o">=</span> <span class="n">sub_ids</span><span class="p">(</span><span class="n">subs</span><span class="o">=</span><span class="n">subs</span><span class="p">)</span>
<span class="c1"># sleep 30 seconds, guarantee all subscribes have subscribed</span>
<span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">30</span><span class="p">)</span>
<span class="nb">print</span> <span class="s2">"regain consciousness"</span>
<span class="n">start_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="k">for</span> <span class="n">each</span> <span class="ow">in</span> <span class="n">xrange</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">times</span><span class="o">+</span><span class="mi">1</span><span class="p">):</span>
<span class="k">if</span> <span class="n">broadcast</span><span class="p">:</span>
<span class="n">socket</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">for</span> <span class="n">each_sub</span> <span class="ow">in</span> <span class="n">sub_list</span><span class="p">:</span>
<span class="n">socket</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">each_sub</span><span class="p">,</span> <span class="n">flags</span><span class="o">=</span><span class="n">zmq</span><span class="o">.</span><span class="n">SNDMORE</span><span class="p">)</span>
<span class="n">socket</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
<span class="n">end_time</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">time</span><span class="p">()</span>
<span class="nb">print</span> <span class="s2">"-------------------------------"</span>
<span class="nb">print</span> <span class="s2">"Exec times: </span><span class="si">%s</span><span class="s2">, Exec time: </span><span class="si">%d</span><span class="s2">ms"</span> <span class="o">%</span><span class="p">(</span><span class="n">times</span><span class="p">,</span> <span class="n">end_time</span> <span class="o">*</span> <span class="mi">1000</span> <span class="o">-</span> <span class="n">start_time</span> <span class="o">*</span> <span class="mi">1000</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="n">subs</span> <span class="o">=</span> <span class="mi">1000</span>
<span class="n">times</span> <span class="o">=</span> <span class="mi">1000</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">==</span> <span class="mi">2</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'unicast'</span><span class="p">:</span>
<span class="n">broadcast</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">broadcast</span> <span class="o">=</span> <span class="kc">True</span>
<span class="nb">print</span> <span class="s2">"broadcast subscriber: </span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span> <span class="n">broadcast</span>
<span class="n">main</span><span class="p">(</span><span class="n">broadcast</span><span class="o">=</span><span class="n">broadcast</span><span class="p">,</span> <span class="n">times</span><span class="o">=</span><span class="n">times</span><span class="p">,</span> <span class="n">subs</span><span class="o">=</span><span class="n">subs</span><span class="p">)</span>
</pre></div>
<p><strong>subscriber.py</strong></p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">zmq</span>
<span class="kn">import</span> <span class="nn">threading</span>
<span class="kn">import</span> <span class="nn">hashlib</span>
<span class="k">def</span> <span class="nf">sub</span><span class="p">(</span><span class="n">pub_uri</span><span class="p">,</span> <span class="n">times</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">idx</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="n">context</span> <span class="o">=</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">()</span>
<span class="n">socket</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">SUB</span><span class="p">)</span>
<span class="n">socket</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">pub_uri</span><span class="p">)</span>
<span class="k">if</span> <span class="n">idx</span><span class="p">:</span>
<span class="n">socket</span><span class="o">.</span><span class="n">setsockopt</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">SUBSCRIBE</span><span class="p">,</span> <span class="n">idx</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">socket</span><span class="o">.</span><span class="n">setsockopt</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">SUBSCRIBE</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
<span class="k">for</span> <span class="n">each</span> <span class="ow">in</span> <span class="n">xrange</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">times</span> <span class="o">+</span> <span class="mi">1</span><span class="p">):</span>
<span class="n">socket</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="n">socket</span><span class="o">.</span><span class="n">close</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">(</span><span class="n">subs</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">times</span><span class="o">=</span><span class="mi">100</span><span class="p">,</span> <span class="n">broadcast</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
<span class="n">pub_uri</span> <span class="o">=</span> <span class="s1">'tcp://salt-master.example.com:5556'</span>
<span class="n">sub_list</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">each_sub</span> <span class="ow">in</span> <span class="n">xrange</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">subs</span><span class="o">+</span><span class="mi">1</span><span class="p">):</span>
<span class="k">if</span> <span class="n">broadcast</span><span class="p">:</span>
<span class="n">sub_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">sub</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">pub_uri</span><span class="p">,),</span> <span class="n">kwargs</span><span class="o">=</span><span class="p">{</span><span class="s1">'times'</span><span class="p">:</span> <span class="n">times</span><span class="p">}))</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">idx</span> <span class="o">=</span> <span class="n">hashlib</span><span class="o">.</span><span class="n">md5</span><span class="p">(</span><span class="nb">str</span><span class="p">(</span><span class="n">each_sub</span><span class="p">))</span><span class="o">.</span><span class="n">hexdigest</span><span class="p">()</span>
<span class="n">sub_list</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">threading</span><span class="o">.</span><span class="n">Thread</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">sub</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">pub_uri</span><span class="p">,),</span> <span class="n">kwargs</span><span class="o">=</span><span class="p">{</span><span class="s1">'times'</span><span class="p">:</span> <span class="n">times</span><span class="p">,</span> <span class="s1">'idx'</span><span class="p">:</span> <span class="n">idx</span><span class="p">}))</span>
<span class="k">for</span> <span class="n">each_sub</span> <span class="ow">in</span> <span class="n">sub_list</span><span class="p">:</span>
<span class="n">each_sub</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="nb">print</span> <span class="s2">"subscriber start ok"</span>
<span class="k">for</span> <span class="n">each_sub</span> <span class="ow">in</span> <span class="n">sub_list</span><span class="p">:</span>
<span class="n">each_sub</span><span class="o">.</span><span class="n">join</span><span class="p">()</span>
<span class="nb">print</span> <span class="s2">"subscriber done"</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="n">subs</span> <span class="o">=</span> <span class="mi">1000</span>
<span class="n">times</span> <span class="o">=</span> <span class="mi">1000</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">)</span> <span class="o">==</span> <span class="mi">2</span> <span class="ow">and</span> <span class="n">sys</span><span class="o">.</span><span class="n">argv</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o">==</span> <span class="s1">'unicast'</span><span class="p">:</span>
<span class="n">broadcast</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">broadcast</span> <span class="o">=</span> <span class="kc">True</span>
<span class="nb">print</span> <span class="s2">"broadcast subscriber: </span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span> <span class="n">broadcast</span>
<span class="n">main</span><span class="p">(</span><span class="n">subs</span><span class="o">=</span><span class="n">subs</span><span class="p">,</span> <span class="n">times</span><span class="o">=</span><span class="n">times</span><span class="p">,</span> <span class="n">broadcast</span><span class="o">=</span><span class="n">broadcast</span><span class="p">)</span>
</pre></div>
<p><strong>测试用例1: 默认的PUB-SUB</strong></p>
<div class="highlight"><pre><span></span>python<span class="w"> </span>publisher.py<span class="w"> </span><span class="c1"># 在Publisher(Master)端进行</span>
python<span class="w"> </span>subscriber.py<span class="w"> </span><span class="c1"># 在Subcriber(Minion-01)端进行</span>
</pre></div>
<p>执行时间为 1588ms</p>
<p><strong>测试用例2: 启用PUB-SUB Message Envelopes</strong></p>
<div class="highlight"><pre><span></span>python<span class="w"> </span>publisher.py<span class="w"> </span>unicast<span class="w"> </span><span class="c1"># 在Publisher(Master)端进行</span>
python<span class="w"> </span>subscriber.py<span class="w"> </span>unicast<span class="w"> </span><span class="c1"># 在Subcriber(Minion-01)端进行</span>
</pre></div>
<p>执行时间为 6786ms</p>
<p>两者相差5s左右, 由于本次测试, 为极端情况(1000次且每次都需要发送到所有subscriber), 增加的成本在可承受范围之内.</p>
</div>
<div class="section" id="qa">
<h3>QA</h3>
<p><strong>Q: 如果只是单边启动zmq_filtering, 是否会影响使用?</strong></p>
<p>A: 这里边有两种情况</p>
<ul class="simple">
<li>Master端配置了zmq_filtering, 对于没有配置zmq_filtering的Minion, 将像以前一样, master端依然会将消息发送给它(不管target是否匹配), 对于已经开启zmq_filtering的minion, 则如果list target不匹配, master则不会发送消息给它.</li>
<li>Master没有配置zmq_filtering, 而Minion进行了配置, 则该Minion收不到任何指令</li>
</ul>
<p><strong>Q: zmq_filtering适用场景</strong></p>
<p>A: 当前zmq_filtering只会匹配list target, 对于其他的target方式, 则采用默认的PUB-SUB. zmq_filtering适用于大规模集群, 但每次执行只是少数主机运行指令的场景.</p>
</div>
</div>
显示Salt进程具体名称2015-01-08T00:00:00+08:002015-01-08T00:00:00+08:00pengyaotag:pengyao.org,2015-01-08:/howto-display-salt-process-name.html<p class="first last"><a class="reference external" href="http://saltstack.com/">Salt</a> 当前已经支持显示具体的进程名, 只需要安装 <a class="reference external" href="https://pypi.python.org/pypi/setproctitle">setproctitle</a> 重启后即可显示Salt进程的具体名称, 便于Debug</p>
<p><a class="reference external" href="http://saltstack.com/">Salt</a> 当前已经支持显示具体的进程名, 只需要安装 <a class="reference external" href="https://pypi.python.org/pypi/setproctitle">setproctitle</a> 重启后即可显示Salt进程的具体名称, 便于Debug</p>
<div class="section" id="section-1">
<h2>环境说明</h2>
<ul class="simple">
<li>操作系统环境: CentOS 6.5,已配置EPEL源</li>
<li>Salt Master/Minion版本: 2014.7.0</li>
</ul>
</div>
<div class="section" id="section-2">
<h2>测试</h2>
<p>安装setproctitle(Master/Minion端均进行)</p>
<div class="highlight"><pre><span></span>yum<span class="w"> </span>-y<span class="w"> </span>install<span class="w"> </span>python-setproctitle
</pre></div>
<p>重启salt</p>
<div class="highlight"><pre><span></span>service<span class="w"> </span>salt-master<span class="w"> </span>restart
service<span class="w"> </span>salt-minion<span class="w"> </span>restart
</pre></div>
<p>查看Master端进程</p>
<div class="highlight"><pre><span></span>ps<span class="w"> </span>ax<span class="w"> </span><span class="p">|</span>grep<span class="w"> </span>salt<span class="w"> </span><span class="p">|</span>grep<span class="w"> </span>-v<span class="w"> </span>salt
</pre></div>
<p>Master端显示如下(同时个人在行尾追加上进程的具体用途):</p>
<pre class="literal-block">
2943 ? S 0:00 /usr/bin/python /usr/bin/salt-master -d ProcessManager # 中心进程管理器
2944 ? S 0:00 /usr/bin/python /usr/bin/salt-master -d _clear_old_jobs # 清除旧的Jobs文件及更新fileserver
2945 ? Sl 0:00 /usr/bin/python /usr/bin/salt-master -d Publisher # 将任务PUB到Minion端
2946 ? Sl 0:00 /usr/bin/python /usr/bin/salt-master -d EventPublisher # Event Publisher进程
2951 ? S 0:00 /usr/bin/python /usr/bin/salt-master -d ReqServer_ProcessManager # ReqServer进程管理器
2952 ? Sl 0:01 /usr/bin/python /usr/bin/salt-master -d MWorker # 劳苦大众, 奋斗在一线的Worker进程
2953 ? Sl 0:01 /usr/bin/python /usr/bin/salt-master -d MWorker # 同楼上
2954 ? Sl 0:01 /usr/bin/python /usr/bin/salt-master -d MWorker
2955 ? Sl 0:01 /usr/bin/python /usr/bin/salt-master -d MWorker
2956 ? Sl 0:01 /usr/bin/python /usr/bin/salt-master -d MWorker
2957 ? Sl 0:00 /usr/bin/python /usr/bin/salt-master -d MWorkerQueue # 将Ret接口(ROUTER)数据转发到Worker(DEALER)
</pre>
<p>执行个任务, 看看Minion端怎么显示(同时个人在行尾追加上进程的具体用途):</p>
<pre class="literal-block">
2003 ? Sl 0:01 /usr/bin/python /usr/bin/salt-minion -d # Minion进程, 接收来自Master端的任务
2069 ? S 0:00 /usr/bin/python /usr/bin/salt-minion -d 20150108034936245247 # 接收到任务后, 会启动名为对应jid的进程进行任务处理及结果反馈
</pre>
<p>这样, 就可以非常清晰的知道Salt的每个进程是做什么用途的, 如果Master/Minion进程异常, 也可以迅速的定位</p>
</div>
基于Salt管理iptables防火墙规则2014-12-17T00:00:00+08:002014-12-17T00:00:00+08:00pengyaotag:pengyao.org,2014-12-17:/managing-firewall-with-salt.html<p class="first last">Salt 2014.7支持pillar merge功能, 尝试基于此完成统一的iptables防火墙规则的管理</p>
<p>Salt 2014.7支持pillar merge功能, 尝试基于该功能, 进行统一的iptables防火墙的管理. 本文采用在iptables INPUT链中增加防火墙规则
(各个服务对应自己的自定义链),同时如果规则中有allow, 则表示该防火墙规则为白名单机制(只允许allow对应的主机访问, 其余均拒绝),
如果没有allow, 则判断是否存在deny, 如果存在, 则执行黑名单机制(只拒绝deny对应的主机, 其余均允许)</p>
<div class="section" id="section-1">
<h2>环境说明</h2>
<ul class="simple">
<li>OS: CentOS 6.5</li>
<li>Salt架构: Master/Minions架构, 版本为2014.7.0</li>
</ul>
<p>由于2014.7.0中iptables模块存在匹配Bug, 导致会不断进行重复配置, 当前develop分支已经修复这一问题(为修复这个问题的思路点赞), 需要进行对Minion进行如下操作:</p>
<div class="highlight"><pre><span></span><span class="c1"># 更新已修复匹配Bug的最新iptables模块</span>
curl<span class="w"> </span>-so<span class="w"> </span>/usr/lib/python2.6/site-packages/salt/modules/iptables.py<span class="w"> </span><span class="se">\</span>
https://raw.githubusercontent.com/saltstack/salt/develop/salt/modules/iptables.py
<span class="c1"># 重启salt-minion</span>
service<span class="w"> </span>salt-minion<span class="w"> </span>restart
</pre></div>
</div>
<div class="section" id="section-2">
<h2>开工</h2>
<div class="section" id="pillar">
<h3>Pillar</h3>
<p>/srv/pillar/sshd/init.sls:</p>
<div class="highlight"><pre><span></span><span class="nt">firewall</span><span class="p">:</span>
<span class="w"> </span><span class="nt">sshd_firewall</span><span class="p">:</span>
<span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">22</span>
<span class="w"> </span><span class="nt">deny</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">192.168.0.0/24</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">8.8.8.8</span>
</pre></div>
<p>/srv/pillar/httpd/init.sls:</p>
<div class="highlight"><pre><span></span><span class="nt">firewall</span><span class="p">:</span>
<span class="w"> </span><span class="nt">httpd_firewall</span><span class="p">:</span>
<span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">80</span>
<span class="w"> </span><span class="nt">allow</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">127.0.0.1</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">192.168.0.0/24</span>
</pre></div>
<p>/srv/pillar/top.sls</p>
<div class="highlight"><pre><span></span><span class="nt">base</span><span class="p">:</span>
<span class="w"> </span><span class="s">'*'</span><span class="p p-Indicator">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">sshd</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">httpd</span>
</pre></div>
<p>获取pillar信息:</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>pillar.item<span class="w"> </span>firewall<span class="w"> </span>--output<span class="o">=</span>yaml
</pre></div>
<p>结果如下:</p>
<div class="highlight"><pre><span></span><span class="nt">minion-01.example.com</span><span class="p">:</span>
<span class="w"> </span><span class="nt">firewall</span><span class="p">:</span>
<span class="w"> </span><span class="nt">httpd_firewall</span><span class="p">:</span>
<span class="w"> </span><span class="nt">allow</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">127.0.0.1</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">192.168.0.0/24</span>
<span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">80</span>
<span class="w"> </span><span class="nt">sshd_firewall</span><span class="p">:</span>
<span class="w"> </span><span class="nt">deny</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">192.168.0.0/24</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">8.8.8.8</span>
<span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">22</span>
</pre></div>
</div>
<div class="section" id="state">
<h3>State</h3>
<p>/srv/salt/iptables/init.sls:</p>
<pre class="literal-block">
{% for eachfw, fw_rule in pillar['firewall'].iteritems() %}
# Add custom chain
{{ eachfw }}-chain:
iptables.chain_present:
- save: True
# Custom chain rules
{% if 'allow' in fw_rule %}
# White Lists
{% for each_allow in fw_rule['allow'] %}
{{ eachfw }}_allow_{{ each_allow }}:
iptables.insert:
- table: filter
- chain: {{ eachfw }}-chain
- position: 1
- source: {{ each_allow }}
- jump: ACCEPT
- require:
- iptables: {{ eachfw }}-chain
- require_in:
- iptables: {{ eachfw }}_deny
- save: True
{% endfor %}
# Deny all
{{ eachfw }}_deny:
iptables.append:
- table: filter
- chain: {{ eachfw }}-chain
- jump: DROP
- save: True
{% elif 'deny' in fw_rule %}
# Black Lists
{% for each_deny in fw_rule['deny'] %}
{{ eachfw }}_deny_{{ each_deny }}:
iptables.insert:
- table: filter
- chain: {{ eachfw }}-chain
- position: 1
- source: {{ each_deny }}
- jump: DROP
- require:
- iptables: {{ eachfw }}-chain
- require_in:
- iptables: {{ eachfw }}_allow
- save: True
{% endfor %}
# Accept all
{{ eachfw }}_allow:
iptables.append:
- table: filter
- chain: {{ eachfw }}-chain
- jump: ACCEPT
- save: True
{% endif %}
# Export traffic to custom chain
{{ eachfw }}-main:
iptables.insert:
- table: filter
- chain: INPUT
- position: 1
- proto: tcp
- dport: {{ fw_rule['port'] }}
- jump: {{ eachfw }}-chain
{% endfor %}
</pre>
<p>应用iptables配置管理:</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>state.sls<span class="w"> </span>iptables
</pre></div>
<p>结果输出如下:</p>
<pre class="literal-block">
minion-01.example.com:
----------
ID: sshd_firewall-chain
Function: iptables.chain_present
Result: True
Comment: iptables sshd_firewall-chain chain is already exist in filter table for ipv4
Started: 07:58:23.325688
Duration: 6.976 ms
Changes:
----------
ID: sshd_firewall_deny_192.168.0.0/24
Function: iptables.insert
Result: True
Comment: iptables rule for sshd_firewall_deny_192.168.0.0/24 already set for ipv4 (--source 192.168.0.0/24 --jump DROP)
Saved iptables rule for sshd_firewall_deny_192.168.0.0/24 to: --source 192.168.0.0/24 --jump DROP for ipv4
Started: 07:58:23.333635
Duration: 46.198 ms
Changes:
----------
ID: sshd_firewall_deny_8.8.8.8
Function: iptables.insert
Result: True
Comment: iptables rule for sshd_firewall_deny_8.8.8.8 already set for ipv4 (--source 8.8.8.8 --jump DROP)
Saved iptables rule for sshd_firewall_deny_8.8.8.8 to: --source 8.8.8.8 --jump DROP for ipv4
Started: 07:58:23.380231
Duration: 47.926 ms
Changes:
----------
ID: sshd_firewall_allow
Function: iptables.append
Result: True
Comment: iptables rule for sshd_firewall_allow already set (/sbin/iptables -t filter -A sshd_firewall-chain --jump ACCEPT) for ipv4
Saved iptables rule for sshd_firewall_allow to: /sbin/iptables -t filter -A sshd_firewall-chain --jump ACCEPT for ipv4
Started: 07:58:23.430386
Duration: 50.731 ms
Changes:
----------
ID: sshd_firewall-main
Function: iptables.insert
Result: True
Comment: iptables rule for sshd_firewall-main already set for ipv4 (-p tcp --dport 22 --jump sshd_firewall-chain)
Started: 07:58:23.481324
Duration: 38.941 ms
Changes:
----------
ID: httpd_firewall-chain
Function: iptables.chain_present
Result: True
Comment: iptables httpd_firewall-chain chain is already exist in filter table for ipv4
Started: 07:58:23.520640
Duration: 9.483 ms
Changes:
----------
ID: httpd_firewall_allow_127.0.0.1
Function: iptables.insert
Result: True
Comment: iptables rule for httpd_firewall_allow_127.0.0.1 already set for ipv4 (--source 127.0.0.1 --jump ACCEPT)
Saved iptables rule for httpd_firewall_allow_127.0.0.1 to: --source 127.0.0.1 --jump ACCEPT for ipv4
Started: 07:58:23.530949
Duration: 48.088 ms
Changes:
----------
ID: httpd_firewall_allow_192.168.0.0/24
Function: iptables.insert
Result: True
Comment: iptables rule for httpd_firewall_allow_192.168.0.0/24 already set for ipv4 (--source 192.168.0.0/24 --jump ACCEPT)
Saved iptables rule for httpd_firewall_allow_192.168.0.0/24 to: --source 192.168.0.0/24 --jump ACCEPT for ipv4
Started: 07:58:23.579515
Duration: 50.945 ms
Changes:
----------
ID: httpd_firewall_deny
Function: iptables.append
Result: True
Comment: iptables rule for httpd_firewall_deny already set (/sbin/iptables -t filter -A httpd_firewall-chain --jump DROP) for ipv4
Saved iptables rule for httpd_firewall_deny to: /sbin/iptables -t filter -A httpd_firewall-chain --jump DROP for ipv4
Started: 07:58:23.631684
Duration: 50.886 ms
Changes:
----------
ID: httpd_firewall-main
Function: iptables.insert
Result: True
Comment: iptables rule for httpd_firewall-main already set for ipv4 (-p tcp --dport 80 --jump httpd_firewall-chain)
Started: 07:58:23.682788
Duration: 44.153 ms
Changes:
Summary
-------------
Succeeded: 10
Failed: 0
-------------
Total states run: 10
</pre>
<p>检查minion端iptables规则</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>cmd.run<span class="w"> </span><span class="s1">'iptables-save'</span>
</pre></div>
<p>结果:</p>
<pre class="literal-block">
minion-01.example.com:
# Generated by iptables-save v1.4.7 on Wed Dec 17 08:01:51 2014
*filter
:INPUT ACCEPT [65:13902]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [79:24034]
:httpd_firewall-chain - [0:0]
:sshd_firewall-chain - [0:0]
-A INPUT -p tcp -m tcp --dport 80 -j httpd_firewall-chain
-A INPUT -p tcp -m tcp --dport 22 -j sshd_firewall-chain
-A httpd_firewall-chain -s 192.168.0.0/24 -j ACCEPT
-A httpd_firewall-chain -s 127.0.0.1/32 -j ACCEPT
-A httpd_firewall-chain -j DROP
-A sshd_firewall-chain -s 8.8.8.8/32 -j DROP
-A sshd_firewall-chain -s 192.168.0.0/24 -j DROP
-A sshd_firewall-chain -j ACCEPT
COMMIT
# Completed on Wed Dec 17 08:01:51 2014
</pre>
<p>达到预期</p>
</div>
</div>
Salt Master外部Job Cache配置2014-11-20T00:00:00+08:002014-11-20T00:00:00+08:00pengyaotag:pengyao.org,2014-11-20:/saltstack-master-external-job-cache.html<p class="first last">SaltStack 2014.7.0对master端的job_cache进行了扩展,可以轻松配置将job执行结果存储到外部系统中.</p>
<p>SaltStack 2014.7之前, Minion端的执行结果想存储在外部系统中, 需要借助returner进行配置. 而returner的工作方式是由minion端直接连接对应的returner, 在分布式环境中由于网络等限制,该方式并不友好.</p>
<p>而在Master端, Job Cache会以文件的形式存储在Master本地磁盘, 对第三方系统并不友好. 基于此, 之前有分享过 <a class="reference external" href="http://pengyao.org/saltstack_master_retuner_over_event_system.html">基于Salt Event系统构建Master端returner</a> , 需要启动另外一个进程, 进行监听Salt Event接口, 并将结果存储在第三方系统中. 刚好看到2014.7.0中master端增加了 <a class="reference external" href="http://docs.saltstack.com/en/latest/ref/configuration/master.html#master-job-cache">master_job_cache</a> 参数, 可以直接外放Job Cache, 就做个测试, 测试下这个功能.</p>
<div class="section" id="section-1">
<h2>环境说明</h2>
<ul class="simple">
<li>Salt Version: <em>2014.7.0</em></li>
<li>OS: CentOS 6.5 X86_64 (with <a class="reference external" href="https://fedoraproject.org/wiki/EPEL">EPEL</a>)</li>
<li>本次测试结果将存储在MySQL中, 为了方便测试, 已在Master本地部署了MySQL Server</li>
</ul>
</div>
<div class="section" id="section-2">
<h2>开工</h2>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">以下操作如非特别注明, 均在Master端进行</p>
</div>
<div class="section" id="section-3">
<h3>前置配置</h3>
<p>安装MySQLdb依赖:</p>
<div class="highlight"><pre><span></span>yum<span class="w"> </span>-y<span class="w"> </span>install<span class="w"> </span>MySQL-python
</pre></div>
<p>配置本次测试需要使用的数据库及用户:</p>
<div class="highlight"><pre><span></span><span class="c1"># 创建salt数据库</span>
mysql<span class="w"> </span>-e<span class="w"> </span><span class="s1">'create database salt'</span>
<span class="c1"># 创建用于连接salt数据库的用户</span>
mysql<span class="w"> </span>-e<span class="w"> </span><span class="s1">'"grant all on salt.* to salt@localhost identified by "salt_pass'</span><span class="p">;</span>
<span class="c1"># 将数据库配置添加至master配置文件中</span>
</pre></div>
<p>创建用于存储Job的数据库表结构:</p>
<pre class="literal-block">
USE `salt`;
--
-- Table structure for table `jids`
--
DROP TABLE IF EXISTS `jids`;
CREATE TABLE `jids` (
`jid` varchar(255) NOT NULL,
`load` mediumtext NOT NULL,
UNIQUE KEY `jid` (`jid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
--
-- Table structure for table `salt_returns`
--
DROP TABLE IF EXISTS `salt_returns`;
CREATE TABLE `salt_returns` (
`fun` varchar(50) NOT NULL,
`jid` varchar(255) NOT NULL,
`return` mediumtext NOT NULL,
`id` varchar(255) NOT NULL,
`success` varchar(10) NOT NULL,
`full_ret` mediumtext NOT NULL,
`alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
KEY `id` (`id`),
KEY `jid` (`jid`),
KEY `fun` (`fun`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
</pre>
</div>
<div class="section" id="master">
<h3>配置Master</h3>
<p>将MySQL连接权限等信息添加到Salt Master配置文件中:</p>
<div class="highlight"><pre><span></span><span class="nb">echo</span><span class="w"> </span>-e<span class="w"> </span><span class="s2">"\n\n# MySQL\nmysql.host: 'localhost'\nmysql.user: 'salt'\nmysql.pass: 'salt_pass'\nmysql.db: 'salt'\nmysql.port: 3306"</span><span class="w"> </span>>><span class="w"> </span>/etc/salt/master
</pre></div>
<p>配置master_job_cache选项, 以使将Job结果存储在MySQL中:</p>
<div class="highlight"><pre><span></span><span class="nb">echo</span><span class="w"> </span>-e<span class="w"> </span><span class="s2">"\n\n# Master Job Cache\nmaster_job_cache: mysql"</span><span class="w"> </span>>><span class="w"> </span>/etc/salt/master
</pre></div>
<p>重启Salt Master, 以使配置生效:</p>
<div class="highlight"><pre><span></span>service<span class="w"> </span>salt-master<span class="w"> </span>restart
</pre></div>
</div>
<div class="section" id="section-4">
<h3>测试</h3>
<p>对主机执行test.ping:</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>test.ping<span class="w"> </span>-v
</pre></div>
<p>输出结果:</p>
<pre class="literal-block">
Executing job with jid 20141120060202308159
-------------------------------------------
minion-01.example.com:
True
</pre>
<p>查询MySQL jids表数据:</p>
<div class="highlight"><pre><span></span>mysql<span class="w"> </span>salt<span class="w"> </span>-e<span class="w"> </span><span class="s1">'select * from jids\G'</span>
</pre></div>
<p>输出结果:</p>
<pre class="literal-block">
*************************** 1. row ***************************
jid: 20141120060202308159
load: {"tgt_type": "glob", "jid": "20141120060202308159", "cmd": "publish", "tgt": "*", "kwargs": {"show_timeout": false, "show_jid": false}, "ret": "", "user": "sudo_vagrant", "arg": [], "fun": "test.ping"}
</pre>
<p>查询MySQL salt_returns表数据:</p>
<div class="highlight"><pre><span></span>mysql<span class="w"> </span>salt<span class="w"> </span>-e<span class="w"> </span><span class="s1">'select * from salt_returns\G'</span>
</pre></div>
<p>输出结果:</p>
<pre class="literal-block">
*************************** 1. row ***************************
fun: test.ping
jid: 20141120060202308159
return: true
id: minion-01.example.com
success: 1
full_ret: {"fun_args": [], "jid": "20141120060202308159", "return": true, "retcode": 0, "success": true, "cmd": "_return", "_stamp": "2014-11-20T06:02:02.533850", "fun": "test.ping", "id": "minion-01.example.com"}
alter_time: 2014-11-20 06:02:02
</pre>
<p>Job执行结果已经按照之前的配置存储到了MySQL中, 达到预期效果</p>
</div>
</div>
salt-broker: 轻量级的Salt Proxy解决方案2014-09-07T00:00:00+08:002014-09-07T00:00:00+08:00pengyaotag:pengyao.org,2014-09-07:/salt-broker-01.html<p class="first last">当前运维系统底层采用 <a class="reference external" href="https://github.com/saltstack/salt">Salt</a> 进行实现, 由于节点分布在全国各地, 存在南北通畅问题, 为了解决这个问题, 之前采用了 <a class="reference external" href="http://docs.saltstack.com/en/latest/topics/topology/syndic.html">Syndic</a> 方案, 在实际使用中发现由于Syndic采用分治机制, 弱化了MasterOfMaster, 在某些网络状况较差的情况下, 会让结果变得不可控. 为了解决该问题, 借鉴ZeroMQ文档, 开发了轻量的Salt Proxy解决方案 <a class="reference external" href="https://github.com/pengyao/salt-broker">salt-broker</a></p>
<div class="section" id="section-1">
<h2>基本简介</h2>
<p>当前运维系统底层采用 <a class="reference external" href="https://github.com/saltstack/salt">Salt</a> 进行实现, 由于节点分布在全国各地, 存在南北通畅问题, 为了解决这个问题, 之前采用了 <a class="reference external" href="http://docs.saltstack.com/en/latest/topics/topology/syndic.html">Syndic</a> 方案, 在实际使用中发现由于Syndic采用分治机制, 弱化了MasterOfMaster, 在某些网络状况较差的情况下, 会让结果变得不可控. 为了解决该问题, 借鉴ZeroMQ文档, 开发了轻量的Salt Proxy解决方案 <a class="reference external" href="https://github.com/pengyao/salt-broker">salt-broker</a></p>
</div>
<div class="section" id="section-2">
<h2>前置阅读</h2>
<ul class="simple">
<li><a class="reference external" href="http://zguide.zeromq.org/page:all#Chapter-Sockets-and-Patterns">0MQ - The Guide: Sockets and Patterns</a></li>
<li><a class="reference external" href="http://pengyao.org/salt-zeromq-01.html">Salt中ZeroMQ那点事</a></li>
<li><a class="reference external" href="http://pengyao.org/salt-syndic-01.html">Salt中Syndic那点事</a></li>
</ul>
</div>
<div class="section" id="section-3">
<h2>环境说明</h2>
<ul class="simple">
<li>CentOS6.4</li>
<li>Salt <a class="reference external" href="https://github.com/saltstack/salt/tree/v2014.1.10/salt">2014.1.10</a> ,默认配置</li>
<li>由于本文为原理解析, 所以采用的代码为最初版的代码, 只描述了其功能逻辑</li>
</ul>
</div>
<div class="section" id="salt-broker">
<h2>为什么会有salt-broker?</h2>
<p>因为采用Syndic, 在网络链路不好的情况下, syndic架构将变得不可控. 详情请访问 <a class="reference external" href="http://pengyao.org/salt-syndic-01.html">Salt中Syndic那点事</a></p>
<p>而由于业务系统采用Salt作为中心调度, 结果不可控将变得非常糟糕. 尝试在syndic基础上进行修改, 没有达到预期. 分析了需求, 其实自己需要的是一个强中心, 轻量级的Salt Proxy解决方案, 所以就有了 <a class="reference external" href="https://github.com/pengyao/salt-broker">salt-broker</a> .</p>
</div>
<div class="section" id="salt-broker-1">
<h2>salt-broker是什么?</h2>
<p>salt-broker是轻量级的Salt proxy解决方案, 只做数据转发, 不做额外的处理. 其工作原理如下:</p>
<div class="section" id="pub-broker">
<h3>PUB Broker</h3>
<p>在Master/Minions结构中, 命令分发采用ZeroMQ PUB/SUB模式, 如下图:</p>
<img alt="" src="https://raw.githubusercontent.com/imatix/zguide/master/images/fig4.png" />
<p>salt-broker中的PUB Broker在中间增加了Forwarder Proxy层, 使架构变成如下:</p>
<img alt="" src="https://raw.githubusercontent.com/imatix/zguide/master/images/fig18.png" />
<p>对应代码如下:</p>
<div class="highlight"><pre><span></span><span class="n">master_pub</span> <span class="o">=</span> <span class="s2">"tcp://</span><span class="si">%s</span><span class="s2">:</span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">master_ip</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">pub_port</span><span class="p">)</span>
<span class="n">context</span> <span class="o">=</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">()</span>
<span class="n">frontend</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">SUB</span><span class="p">)</span>
<span class="n">frontend</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">master_pub</span><span class="p">)</span>
<span class="n">backend</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">PUB</span><span class="p">)</span>
<span class="n">backend</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="s2">"tcp://0.0.0.0:</span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span><span class="bp">self</span><span class="o">.</span><span class="n">pub_port</span><span class="p">)</span>
<span class="n">frontend</span><span class="o">.</span><span class="n">setsockopt</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">SUBSCRIBE</span><span class="p">,</span> <span class="sa">b</span><span class="s1">''</span><span class="p">)</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">message</span> <span class="o">=</span> <span class="n">frontend</span><span class="o">.</span><span class="n">recv_multipart</span><span class="p">()</span>
<span class="n">backend</span><span class="o">.</span><span class="n">send_multipart</span><span class="p">(</span><span class="n">message</span><span class="p">)</span>
</pre></div>
<p>本地建立PUB接口, 并连接Master的SUB接口, 订阅来自于Master的消息, 接收到后立马发送到本地的PUB接口. 由于Minions上指定的Master地址为salt-broker所在的地址, 所以Minions能够接受到该消息.</p>
</div>
<div class="section" id="ret-broker">
<h3>Ret Broker</h3>
<p>在Master/Minions结构中, 认证,文件服务,结果收集等采用ZeroMQ ROUTER/REQ模式, 如下图:</p>
<img alt="" src="https://raw.githubusercontent.com/imatix/zguide/master/images/fig20.png" />
<p>salt-broker中的Ret Broker在原来的REQ/ROUTER之间再增加了一层ROUTER/DEALER Proxy层. 对应的代码如下:</p>
<div class="highlight"><pre><span></span><span class="n">master_ret</span> <span class="o">=</span> <span class="s2">"tcp://</span><span class="si">%s</span><span class="s2">:</span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">master_ip</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">ret_port</span><span class="p">)</span>
<span class="n">context</span> <span class="o">=</span> <span class="n">zmq</span><span class="o">.</span><span class="n">Context</span><span class="p">()</span>
<span class="n">frontend</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">ROUTER</span><span class="p">)</span>
<span class="n">frontend</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="s2">"tcp://0.0.0.0:</span><span class="si">%s</span><span class="s2">"</span> <span class="o">%</span><span class="bp">self</span><span class="o">.</span><span class="n">ret_port</span><span class="p">)</span>
<span class="n">backend</span> <span class="o">=</span> <span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">DEALER</span><span class="p">)</span>
<span class="n">backend</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">master_ret</span><span class="p">)</span>
<span class="n">zmq</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">QUEUE</span><span class="p">,</span> <span class="n">frontend</span><span class="p">,</span> <span class="n">backend</span><span class="p">)</span>
</pre></div>
<p>本地建立ROUTER接口, 接收来自于Minions的REQ请求; 同时本地建立DEALER接口, 连接Master的Router接口, 将接收到的数据发送给远端的Master ROUTER接口.</p>
</div>
<div class="section" id="broker-vs-syndic">
<h3>Broker VS Syndic</h3>
<p>salt-broker与syndic一样, 都支持多层级架构. salt-broker相对于syndic, 更为轻量级, 只做数据转发. 在超大规模场景下, salt-broker并不能有效的降低master的压力, 而syndic能够降低.</p>
<p>syndic本地会维护auth及文件服务系统, 而broker会将所有请求转发给Master, 即所有的Minions的最终管理都是在Master上. 由于所有管理权均在Master上, salt-broker能够解决掉之前Syndic在网络不稳定时的不可控问题.</p>
</div>
</div>
<div class="section" id="salt-broker-2">
<h2>如何使用salt-broker?</h2>
<div class="section" id="section-4">
<h3>全新安装</h3>
<ol class="arabic simple">
<li>安装salt(需提前配置EPEL)</li>
</ol>
<div class="highlight"><pre><span></span>yum<span class="w"> </span>-y<span class="w"> </span>install<span class="w"> </span>salt
</pre></div>
<ol class="arabic simple" start="2">
<li>安装salt-broker</li>
</ol>
<div class="highlight"><pre><span></span>pip<span class="w"> </span>install<span class="w"> </span>salt-broker
wget<span class="w"> </span>https://raw.githubusercontent.com/pengyao/salt-broker/master/pkg/rpm/salt-broker<span class="w"> </span>-O<span class="w"> </span>/etc/rc.d/init.d/salt-broker
chmod<span class="w"> </span>+x<span class="w"> </span>/etc/rc.d/init.d/salt-broker
</pre></div>
<ol class="arabic simple" start="3">
<li>配置salt-broker</li>
</ol>
<p><em>/etc/salt/broker</em></p>
<div class="highlight"><pre><span></span><span class="nt">master</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">master_ip</span>
</pre></div>
<ol class="arabic simple" start="4">
<li>启动salt-broker</li>
</ol>
<div class="highlight"><pre><span></span>service<span class="w"> </span>salt-broker<span class="w"> </span>start
chkconfig<span class="w"> </span>salt-broker<span class="w"> </span>on
</pre></div>
<ol class="arabic simple" start="5">
<li>启动完毕后, 需要将minions配置文件中的master配置为salt-broker所在的主机, 同时重启minions服务</li>
<li>在Master接收minions的key(如果之前已accept, 无需操作本步骤)</li>
</ol>
</div>
<div class="section" id="syndic">
<h3>在Syndic主机上安装</h3>
<ol class="arabic simple">
<li>关于Syndic主机上的syndic及master服务</li>
</ol>
<div class="highlight"><pre><span></span>service<span class="w"> </span>salt-syndic<span class="w"> </span>stop
service<span class="w"> </span>salt-master<span class="w"> </span>stop
chkconfig<span class="w"> </span>salt-syndic<span class="w"> </span>off
chkconfig<span class="w"> </span>salt-master<span class="w"> </span>off
</pre></div>
<ol class="arabic simple" start="2">
<li>安装salt-broker</li>
</ol>
<div class="highlight"><pre><span></span>pip<span class="w"> </span>install<span class="w"> </span>salt-broker
wget<span class="w"> </span>https://raw.githubusercontent.com/pengyao/salt-broker/master/pkg/rpm/salt-broker<span class="w"> </span>-O<span class="w"> </span>/etc/rc.d/init.d/salt-broker
chmod<span class="w"> </span>+x<span class="w"> </span>/etc/rc.d/init.d/salt-broker
</pre></div>
<ol class="arabic simple" start="3">
<li>配置salt-broker</li>
</ol>
<p><em>/etc/salt/broker</em></p>
<div class="highlight"><pre><span></span><span class="nt">master</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">master_ip</span>
</pre></div>
<ol class="arabic simple" start="4">
<li>启动salt-broker</li>
</ol>
<div class="highlight"><pre><span></span>service<span class="w"> </span>salt-broker<span class="w"> </span>start
chkconfig<span class="w"> </span>salt-broker
</pre></div>
<ol class="arabic simple" start="5">
<li>启动完毕后, 需要将原syndic下的minions配置文件中的master配置为salt-broker所在的主机, 并将/etc/salt/pki/minion/minion_master.pub删掉, 然后重启minion服务</li>
<li>在Master接收minions的key(如果之前已accept, 无需操作本步骤)</li>
</ol>
</div>
<div class="section" id="section-5">
<h3>其他注意事项</h3>
<p>默认配置中, 使用的是pub(4505)及ret(4506)端口, 如果master端口并非该端口, 需要在/etc/salt/broker配置文件中增加:</p>
<div class="highlight"><pre><span></span><span class="nt">ret_port</span><span class="p">:</span><span class="w"> </span><span class="s">'new_ret_port'</span>
<span class="nt">publish_port</span><span class="p">:</span><span class="w"> </span><span class="s">'new_publish_port'</span>
</pre></div>
<p>更改后并重启salt-broker服务.</p>
</div>
</div>
Salt中Syndic那点事2014-09-07T00:00:00+08:002014-09-07T00:00:00+08:00pengyaotag:pengyao.org,2014-09-07:/salt-syndic-01.html<p class="first last"><a class="reference external" href="https://github.com/saltstack/salt">Salt</a> 在 <a class="reference external" href="http://docs.saltstack.com/en/latest/topics/releases/0.9.0.html">0.9.0版本</a> 中增加了 <a class="reference external" href="http://docs.saltstack.com/en/latest/topics/topology/syndic.html">Syndic</a> 特性. 通过Syndic, 可以快速构建出多层级的Salt拓扑, 使Salt变得更灵活. 那么Syndic是如何工作的? 当前有哪些优势和局限哪?</p>
<div class="section" id="section-1">
<h2>基本简介</h2>
<p><a class="reference external" href="https://github.com/saltstack/salt">Salt</a> 在 <a class="reference external" href="http://docs.saltstack.com/en/latest/topics/releases/0.9.0.html">0.9.0版本</a> 中增加了 <a class="reference external" href="http://docs.saltstack.com/en/latest/topics/topology/syndic.html">Syndic</a> 特性. Syndic建立在中心Master和Minions之间, 并允许多层分级Syndic, 使Salt拓扑可以变得更为灵活. 那么Syndic是如何工作的? 当前有哪些优势和局限哪?</p>
</div>
<div class="section" id="section-2">
<h2>前置阅读</h2>
<ul class="simple">
<li><a class="reference external" href="http://zguide.zeromq.org/page:all#Chapter-Sockets-and-Patterns">0MQ - The Guide: Sockets and Patterns</a></li>
<li><a class="reference external" href="http://pengyao.org/salt-zeromq-01.html">Salt中ZeroMQ那点事</a></li>
</ul>
</div>
<div class="section" id="section-3">
<h2>环境说明</h2>
<ul class="simple">
<li>CentOS6.4</li>
<li>Salt <a class="reference external" href="https://github.com/saltstack/salt/tree/v2014.1.10/salt">2014.1.10</a> ,默认配置</li>
</ul>
</div>
<div class="section" id="syndic">
<h2>安装配置Syndic</h2>
<div class="section" id="syndic-1">
<h3>安装Syndic</h3>
<p>Syndic是Master的一个小组件, 位于salt-master软件包中, 安装salt-master时就安装了syndic</p>
<div class="highlight"><pre><span></span>yum<span class="w"> </span>-y<span class="w"> </span>install<span class="w"> </span>salt-master
</pre></div>
</div>
<div class="section" id="syndic-2">
<h3>配置Syndic</h3>
<p>默认Syndic的配置文件位于Master配置文件中( <em>/etc/salt/master</em> ), 需要配置的参数也非常简单:</p>
<blockquote>
<ul class="simple">
<li>syndic_master: 配置MasterOfMaster的IP地址</li>
<li>syndic_master_port: 配置MasterOfMaster的ret_port(默认为4506)</li>
<li>syndic_log_file: 指定syndic日志文件(绝对路径)</li>
<li>syndic_pidfile: 指定syndic pid文件(绝对路径)</li>
</ul>
</blockquote>
<p>同时MasterOfMaster需要将 <em>order_masters</em> 选项设置为 <em>True</em>, 为了使配置生效, 需要对其进行重启.</p>
</div>
<div class="section" id="syndic-3">
<h3>运行Syndic</h3>
<div class="highlight"><pre><span></span>chkconfig<span class="w"> </span>salt-master<span class="w"> </span>on
chkconfig<span class="w"> </span>salt-syndic<span class="w"> </span>on
service<span class="w"> </span>salt-master<span class="w"> </span>restart
service<span class="w"> </span>salt-syndic<span class="w"> </span>restart
</pre></div>
</div>
</div>
<div class="section" id="syndic-4">
<h2>Syndic是如何工作的?</h2>
<p>Syndic本质上是一个特殊的Minion, 这不, 其代码就位于 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/minion.py#L1516">minion.py</a> 中.</p>
<p>Syndic需要在本地运行Master, 并将需要管理的Minions的master指向syndic所在的主机. Syndic是这么来工作的:</p>
<ol class="arabic simple">
<li>冒充Minion, 建立连接MasterOfMaster PUB接口的SUB接口, 订阅所有来自MasterOfMaster下发的任务</li>
</ol>
<div class="highlight"><pre><span></span><span class="bp">self</span><span class="o">.</span><span class="n">socket</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">context</span><span class="o">.</span><span class="n">socket</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">SUB</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">socket</span><span class="o">.</span><span class="n">setsockopt</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">SUBSCRIBE</span><span class="p">,</span> <span class="s1">''</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">socket</span><span class="o">.</span><span class="n">setsockopt</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">IDENTITY</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">[</span><span class="s1">'id'</span><span class="p">])</span>
<span class="bp">self</span><span class="o">.</span><span class="n">socket</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">master_pub</span><span class="p">)</span>
</pre></div>
<ol class="arabic simple" start="2">
<li>接收到MasterOfMaster下发的数据后, 首先进行解密, 解密完成后, 将其扔到本地的Master接口上进行二次下发:</li>
</ol>
<div class="highlight"><pre><span></span><span class="n">payload</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">serial</span><span class="o">.</span><span class="n">loads</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">socket</span><span class="o">.</span><span class="n">recv</span><span class="p">())</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_handle_payload</span><span class="p">(</span><span class="n">payload</span><span class="p">)</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="c1"># Send out the publication</span>
<span class="bp">self</span><span class="o">.</span><span class="n">local</span><span class="o">.</span><span class="n">pub</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s1">'tgt'</span><span class="p">],</span>
<span class="n">data</span><span class="p">[</span><span class="s1">'fun'</span><span class="p">],</span>
<span class="n">data</span><span class="p">[</span><span class="s1">'arg'</span><span class="p">],</span>
<span class="n">data</span><span class="p">[</span><span class="s1">'tgt_type'</span><span class="p">],</span>
<span class="n">data</span><span class="p">[</span><span class="s1">'ret'</span><span class="p">],</span>
<span class="n">data</span><span class="p">[</span><span class="s1">'jid'</span><span class="p">],</span>
<span class="n">data</span><span class="p">[</span><span class="s1">'to'</span><span class="p">])</span>
</pre></div>
<ol class="arabic simple" start="3">
<li>在2中进行的二次下发之后, 监听本地event接口, 获取旗下Minions的返回:</li>
</ol>
<div class="highlight"><pre><span></span><span class="n">event</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">local</span><span class="o">.</span><span class="n">event</span><span class="o">.</span><span class="n">get_event</span><span class="p">(</span><span class="mf">0.5</span><span class="p">,</span> <span class="n">full</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</pre></div>
<p>并将返回发送给MasterOfMaster Ret接口</p>
<div class="highlight"><pre><span></span><span class="bp">self</span><span class="o">.</span><span class="n">_return_pub</span><span class="p">(</span><span class="n">jids</span><span class="p">[</span><span class="n">jid</span><span class="p">],</span> <span class="s1">'_syndic_return'</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="syndic-5">
<h2>Syndic的优势和局限</h2>
<div class="section" id="section-4">
<h3>优势</h3>
<ol class="arabic simple">
<li>通过Syndic, 可以建立多层级的Salt拓扑, Syndic下的Minions即可通过Syndic所在的Master进行管理, 也可以通过MasterOfMaster及更高层级的Master进行管理, 架构变得异常灵活.</li>
<li>由于Syndic只订阅MasterOfMaster下发下来的任务, 对于文件服务等, Syndic本地需要进行配置,可以有效的降低MasterOfMaster的负载</li>
</ol>
</div>
<div class="section" id="section-5">
<h3>局限</h3>
<p>由于Syndic弱化了MasterOfMaster, 采用区域自治方法. 在某些应用场景下, 会有局限性:</p>
<ol class="arabic simple">
<li>优势2中的优势, 也带来了局限, 需要保证Syndic上的file_roots及pillar_roots与MasterOfMaster是一致的. 为了解决这个问题, 我们在使用Syndic时采用了 <a class="reference external" href="http://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html">gitfs backend</a></li>
<li>由于Syndic管理了旗下Minions的认证, 使MasterOfMaster并不知道有多少Syndic主机, Syndic下边有多少Minions. 在MasterOfMaster及更高层级的Master上使用salt命令行下发远程执行命令时, 如果Syndic此时与MasterOfMaster网络抖动, 导致没有收到消息或延迟收到消息, MasterOfMaster并不知情; Syndic并没有返回结果或延迟返回结果, MasterOfMaster并不能感知到, 会导致结果不完整. 如果没有其他验证机制, 将变得不可控. 官方提供的解决方案是增大 <em>syndic_wait</em> 选项, 但个人认为只能缓解,并不能根治本问题.</li>
</ol>
</div>
</div>
Salt中ZeroMQ那点事2014-09-06T00:00:00+08:002014-09-06T00:00:00+08:00pengyaotag:pengyao.org,2014-09-06:/salt-zeromq-01.html<p class="first last"><a class="reference external" href="https://github.com/saltstack/salt">Salt</a> 底层网络架构采用 <a class="reference external" href="http://zeromq.org/">ZeroMQ</a> 进行实现(2014.1及之前版本, 从2014.7起, Salt新增 <a class="reference external" href="https://github.com/saltstack/raet">RAET</a> ), 那么Salt都使用了ZeroMQ哪些模式? 各个组件间又是如何协作的?</p>
<div class="section" id="section-1">
<h2>基本简介</h2>
<p><a class="reference external" href="https://github.com/saltstack/salt">Salt</a> 底层网络架构采用 <a class="reference external" href="http://zeromq.org/">ZeroMQ</a> 进行实现(2014.1及之前版本, 从2014.7起, Salt新增 <a class="reference external" href="https://github.com/saltstack/raet">RAET</a> ), <a class="reference external" href="http://docs.saltstack.com/en/latest/topics/index.html#building-on-proven-technology">官方手册</a> 有简短描述. 那么今天就一窥在Salt内部使用了哪些 <a class="reference external" href="http://zeromq.org/">ZeroMQ</a> pattern? 各个组件间又是如何协作的哪?</p>
</div>
<div class="section" id="section-2">
<h2>前置阅读</h2>
<ul class="simple">
<li><a class="reference external" href="http://zguide.zeromq.org/page:all#Chapter-Sockets-and-Patterns">0MQ - The Guide: Sockets and Patterns</a></li>
</ul>
</div>
<div class="section" id="section-3">
<h2>环境说明</h2>
<ul class="simple">
<li>CentOS6.4</li>
<li>Salt <a class="reference external" href="https://github.com/saltstack/salt/tree/v2014.1.10/salt">2014.1.10</a> ,默认配置</li>
</ul>
</div>
<div class="section" id="saltzeromq-patterns">
<h2>Salt中的ZeroMQ patterns</h2>
<div class="section" id="salt-master">
<h3>Salt Master</h3>
<p>Salt Master为Salt中心管控节点. 为Salt环境提供命令下发, 文件, 结果收集等服务.</p>
<p>在Master启动时, 首先启动名为 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/master.py#L452">ReqServer</a> , <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/master.py#L584">ReqServer在初始化</a> 时, 立即创建如下ZeroMQ patterns:</p>
<blockquote>
<ul class="simple">
<li><em>clients</em>:<ul>
<li>ZeroMQ pattern: zmq.ROUTER</li>
<li>listen地址: <a class="reference external" href="tcp://0.0.0.0:4506">tcp://0.0.0.0:4506</a></li>
<li>listen方式: bind</li>
<li>作用: Salt Master Ret接口, 支持认证(auth), 文件服务, 结果收集等功能</li>
</ul>
</li>
<li><em>workers</em>:<ul>
<li>ZeroMQ pattern: zmq.DEALER</li>
<li>listen地址: ipc:///var/run/salt/master/workers.ipc</li>
<li>listen方式: bind</li>
<li>作用: Salt Master任务处理进程接口</li>
</ul>
</li>
</ul>
</blockquote>
<p>同时clients与workers, 建立了一个 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/master.py#L635">zeromq.device</a> :</p>
<div class="highlight"><pre><span></span><span class="n">zmq</span><span class="o">.</span><span class="n">device</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">QUEUE</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">clients</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">workers</span><span class="p">)</span>
</pre></div>
<p>通过zmq.device, 实现了clients接收到请求后, 转发到workers进程接口上进行处理</p>
<p>接下来, Master会启动 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/master.py#L635">Publisher</a> , 立即创建了如下ZeroMQ patterns:</p>
<blockquote>
<ul class="simple">
<li><em>pub</em>:<ul>
<li>ZeroMQ pattern: zmq.PUB</li>
<li>listen地址: <a class="reference external" href="tcp://0.0.0.0:4505">tcp://0.0.0.0:4505</a></li>
<li>listen方式: bind</li>
<li>作用: Salt Master pub接口, 提供远程执行命令发送功能</li>
</ul>
</li>
<li><em>pull</em>:<ul>
<li>ZeroMQ pattern: zmq.PULL</li>
<li>listen地址: ipc:///var/run/salt/master/publish_pull.ipc</li>
<li>listen方式: bind</li>
<li>作用: Salt Master远程执行命令pull接口</li>
</ul>
</li>
</ul>
</blockquote>
<p>pull接口在接收到数据后, 会将数据从pub接口上进行发送:</p>
<div class="highlight"><pre><span></span><span class="n">package</span> <span class="o">=</span> <span class="n">pull_sock</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="n">pub_sock</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">package</span><span class="p">)</span>
</pre></div>
<p>接下来, Master启动 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/utils/event.py#L430">EventPublisher</a>, 以实现Event BUS, 创建了如下ZeroMQ patterns:</p>
<blockquote>
<ul class="simple">
<li><em>epub</em>:<ul>
<li>ZeroMQ pattern: zmq.PUB</li>
<li>listen地址: ipc:///var/run/salt/master/master_event_pub.ipc</li>
<li>listen方式: bind</li>
<li>作用: Salt Master event pub接口, 以方便其他或第三方应用订阅event bus上的event</li>
</ul>
</li>
<li><em>epull</em>:<ul>
<li>ZeroMQ pattern: zmq.PULL</li>
<li>listen地址: ipc:///var/run/salt/master/master_event_pull.ipc</li>
<li>listen方式: bind</li>
<li>作用: Salt Master event pull接口</li>
</ul>
</li>
</ul>
</blockquote>
<p>同时epull接口在收到包时, 会将数据在pub接口上进行发送:</p>
<div class="highlight"><pre><span></span><span class="n">package</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">epull_sock</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">epub_sock</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">package</span><span class="p">)</span>
</pre></div>
<p>在启动EventPublisher之后, Salt Master会继续启动Halite, Reactor系统, 该部分暂不描述. 随后, Salt会启动多个Work进程(默认是5, 在规模较大的环境中, 建议增加配置文件中的 <em>worker_threads</em> 数目来增加该类进程)来进行任务处理, 每个Worker进程会创建如下ZeroMQ patterns:</p>
<blockquote>
<ul class="simple">
<li><em>socket</em><ul>
<li>ZeroMQ pattern: zmq.REP</li>
<li>listen地址: ipc:///var/run/salt/master/workers.ipc</li>
<li>listen方式: connect</li>
<li>作用: Salt Master任务处理进程, 处理验证Minion, 获取Master配置, Mine, pillar, fileserver文件获取, minion event fire到master的event接口, 收集minions的返回结果等任务</li>
</ul>
</li>
</ul>
</blockquote>
</div>
<div class="section" id="salt-minion">
<h3>Salt Minion</h3>
<p>Salt Minion为Salt环境操作节点, 远程命令从Master发送过来后, 会在该主机上进行执行并将结果返回给Master.</p>
<p>Salt <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/minion.py#L524">Minion</a> 在启动时从配置文件中获取Master的地址, 如果为域名, 则进行解析. 解析完毕后, 会连接Master的Ret接口进行key认证. 认证通过, 会获取到master的 <em>publish_port</em> , 这就是为什么在Minion的配置文件中只需要指定Minion的 <em>ret_port</em> (对应minion配置文件中的master_port) 即可.</p>
<p>在获取到master的publish_port(默认为4505)之后, 会建立minion本地的Event接口:</p>
<blockquote>
<ul class="simple">
<li><em>epub</em>:<ul>
<li>ZeroMQ pattern: zmq.PUB</li>
<li>listen地址: ipc:///var/run/salt/minion/minion_event_{id_hash}_pub.ipc</li>
<li>listen方式: bind</li>
<li>作用: Salt Minion event pub接口, 以便其他或第三方应用通过该event bus获取event信息</li>
</ul>
</li>
<li><em>epull</em>:<ul>
<li>ZeroMQ pattern: zmq.PULL</li>
<li>listen地址: ipc:///var/run/salt/minion/minion_event_{id_hash}_pull.ipc</li>
<li>listen方式: bind</li>
<li>作用: Salt Minion event pull接口</li>
</ul>
</li>
</ul>
</blockquote>
<p>epull接口在接收到数据后, 会检查是否需要处理, 如果需要处理, 则进行执行. 随后将该数据包传送到epub接口:</p>
<div class="highlight"><pre><span></span><span class="c1"># Check the event system</span>
<span class="k">if</span> <span class="n">socks</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">epull_sock</span><span class="p">)</span> <span class="o">==</span> <span class="n">zmq</span><span class="o">.</span><span class="n">POLLIN</span><span class="p">:</span>
<span class="n">package</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">epull_sock</span><span class="o">.</span><span class="n">recv</span><span class="p">(</span><span class="n">zmq</span><span class="o">.</span><span class="n">NOBLOCK</span><span class="p">)</span>
<span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Handling event </span><span class="si">%r</span><span class="s2">"</span><span class="p">,</span> <span class="n">package</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="k">if</span> <span class="n">package</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'module_refresh'</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">module_refresh</span><span class="p">()</span>
<span class="k">elif</span> <span class="n">package</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'pillar_refresh'</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">pillar_refresh</span><span class="p">()</span>
<span class="k">elif</span> <span class="n">package</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'grains_refresh'</span><span class="p">):</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">grains_cache</span> <span class="o">!=</span> <span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">[</span><span class="s1">'grains'</span><span class="p">]:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">pillar_refresh</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">grains_cache</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">[</span><span class="s1">'grains'</span><span class="p">]</span>
<span class="k">elif</span> <span class="n">package</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'fire_master'</span><span class="p">):</span>
<span class="n">tag</span><span class="p">,</span> <span class="n">data</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">event</span><span class="o">.</span><span class="n">MinionEvent</span><span class="o">.</span><span class="n">unpack</span><span class="p">(</span><span class="n">package</span><span class="p">)</span>
<span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Forwarding master event tag=</span><span class="si">{tag}</span><span class="s2">"</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">tag</span><span class="o">=</span><span class="n">data</span><span class="p">[</span><span class="s1">'tag'</span><span class="p">]))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_fire_master</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="s1">'data'</span><span class="p">],</span> <span class="n">data</span><span class="p">[</span><span class="s1">'tag'</span><span class="p">],</span> <span class="n">data</span><span class="p">[</span><span class="s1">'events'</span><span class="p">],</span> <span class="n">data</span><span class="p">[</span><span class="s1">'pretag'</span><span class="p">])</span>
<span class="bp">self</span><span class="o">.</span><span class="n">epub_sock</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">package</span><span class="p">)</span>
<span class="k">except</span> <span class="ne">Exception</span><span class="p">:</span>
<span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s2">"Exception while handling events"</span><span class="p">,</span> <span class="n">exc_info</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
</pre></div>
<p>在event接口建立完毕后, 会建立如下ZeroMQ pattern:</p>
<blockquote>
<ul class="simple">
<li><em>socket</em>:<ul>
<li>ZeroMQ pattern: zmq.SUB</li>
<li>listen地址: <a class="reference external" href="tcp:/">tcp:/</a>/{master_ip}:4505</li>
<li>listen方式: connect</li>
<li>作用: 订阅来自Master pub接口的任务</li>
</ul>
</li>
</ul>
</blockquote>
<p>由于远程执行命令的发送, 是通过ZeroMQ PUB/SUB pattern进行建立的, 即当master下发操作指令时, 所有的minion均可以接收到, 然后minion会检查本机是否target match, 如果match, 则进行执行.执行完毕后, 会通过 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/payload.py#L159">SREQ</a> 发送到Master的Ret接口, 期间会创建如下ZeroMQ pattern:</p>
<blockquote>
<ul class="simple">
<li><em>socket</em>:<ul>
<li>ZeroMQ pattern: zmq.REQ</li>
<li>listen地址: <a class="reference external" href="tcp:/">tcp:/</a>/{master_ip}:4506</li>
<li>listen方式: connect</li>
<li>作用: 将执行结果发送给Master</li>
</ul>
</li>
</ul>
</blockquote>
<p>更多关于Minion如何来执行任务, 请访问: <a class="reference external" href="http://devopstarter.info/yuan-ma-jie-du-saltstackyun-xing-ji-zhi-zhi-job-runtime/">http://devopstarter.info/yuan-ma-jie-du-saltstackyun-xing-ji-zhi-zhi-job-runtime/</a></p>
</div>
<div class="section" id="salt">
<h3>Salt</h3>
<p>Salt Master与Salt Minion建立了对应的ZeroMQ pattern, 那么当一个远程执行指令下发下去, 其数据流向是怎么个流程哪? 以执行test.ping为例:</p>
<ol class="arabic simple">
<li>在master端bash下, 执行:</li>
</ol>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>test.ping
</pre></div>
<p>其对应的 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/scripts.py#L126">python执行</a> 是:</p>
<div class="highlight"><pre><span></span><span class="n">client</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">cli</span><span class="o">.</span><span class="n">SaltCMD</span><span class="p">()</span>
<span class="n">client</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</pre></div>
<p>在内部, 又是调用:</p>
<div class="highlight"><pre><span></span><span class="n">local</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">LocalClient</span><span class="p">()</span>
<span class="n">cmd_fun</span> <span class="o">=</span> <span class="n">local</span><span class="o">.</span><span class="n">cmd_cli</span><span class="p">()</span>
<span class="k">for</span> <span class="n">full_ret</span> <span class="ow">in</span> <span class="n">cmd_func</span><span class="p">(</span><span class="n">kwargs</span><span class="p">):</span>
<span class="n">ret</span><span class="p">,</span> <span class="n">out</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_format_ret</span><span class="p">(</span><span class="n">full_ret</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_output_ret</span><span class="p">(</span><span class="n">ret</span><span class="p">,</span> <span class="n">out</span><span class="p">)</span>
</pre></div>
<ol class="arabic simple" start="2">
<li>在 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/client/__init__.py#L77">LocalClient</a> 对象初始化时, 会创建用于对发送的数据进行序列化的 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/client/__init__.py#L77">Serial</a> 对象, 及 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/utils/event.py#L406">MasterEvent</a> 对象. MasterEvent对象会创建如下ZeroMQ pattern:</li>
</ol>
<blockquote>
<ul class="simple">
<li><em>sub</em>:<ul>
<li>ZeroMQ pattern: zmq.SUB</li>
<li>listen地址: ipc:///var/run/salt/master/master_event_pub.ipc</li>
<li>listen方式: connect</li>
<li>作用: 用于订阅来自于Master event pub接口的数据</li>
</ul>
</li>
</ul>
</blockquote>
<ol class="arabic" start="3">
<li><p class="first"><a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/client/__init__.py#L524">cmd_cli</a> 在执行时, 会首先通过 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/client/__init__.py#L234">run_job</a> 将操作指令封装成如下内容:</p>
<blockquote>
<p>{'tgt_type': 'glob', 'jid': '', 'key': 'LCkViTMgqKBqb5ooG8kznznztLYPsWR1xdTYnAz9udkU9/Lla32yDvUmVKLPaUNSMtbWdBoQPIs=', 'tgt': '*', 'arg': [], 'fun': 'test.ping', 'kwargs': {'show_timeout': False}, 'cmd': 'publish', 'ret': '', 'user': 'root'}</p>
</blockquote>
</li>
</ol>
<p>将发送到本地master的Ret接口, 期间会创建如下ZeroMQ pattern:</p>
<blockquote>
<ul class="simple">
<li><em>socket</em>:<ul>
<li>ZeroMQ pattern: zmq.REQ</li>
<li>listen地址: <a class="reference external" href="tcp://127.0.0.1:4506">tcp://127.0.0.1:4506</a></li>
<li>listen方式: connect</li>
<li>作用: 将封装后的指令发送到Master Ret接口</li>
</ul>
</li>
</ul>
</blockquote>
<ol class="arabic simple" start="4">
<li>Master Ret接口接收到3中发送的数据后, 会通过chminions.check_minions获取本次需要哪些minions执行, 并产生jid, 然后在master event接口上进行fire_event操作, 之后对数据使用master私钥(master.pem)进行签名, 然后创建如下ZeroMQ pattern:</li>
</ol>
<blockquote>
<ul class="simple">
<li><em>pub_socket</em>:<ul>
<li>ZeroMQ pattern: zmq.PUSH</li>
<li>listen地址: ipc:///var/run/salt/master/publish_pull.ipc</li>
<li>listen方式: connect</li>
<li>作用: 将指令传送到Master Pull接口</li>
</ul>
</li>
</ul>
</blockquote>
<p>Master Pull接口接收到数据后, 会迅速的在Master Pub接口上发送将之前收到的数据</p>
<p>同时将jid及minions封装后的结果返回给3, 3中cmd_cli获取到数据后, 调用 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/client/__init__.py#L1142">get_cli_event_returns</a> ,监听Master端的Event bus, 过滤出本次任务jid所对应的event, 用来获取执行结果</p>
<ol class="arabic simple" start="5">
<li>此时Minion通过PUB/SUB, 即可收到来自于Master Pub接口的消息. Minion接收到消息后, 会首先通过本地的master pub_key(minion_master.pub)进行解密, 已确保消息来自于Master. 解密完成后, 本地进行target匹配, 如果匹配上, 表示需要执行, 派生出一个新的进程进行执行. 反之则直接忽略.</li>
<li>Minion执行完毕后, 会通过 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/minion.py#L938">_return_pub</a> 将封装后的结果通过AES加密发送到Master的Ret接口</li>
<li>Master Ret接收到6中发送的数据后, 会进行AES解密, 然后通过 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.10/salt/master.py#L1354">_return</a>, 首先将解密后的数据在本地event接口上进行fire_event, 并将结果存储在master本地.</li>
<li>由于7中进行fire_event, 此时4中的get_cli_event_returns即可捕捉到, 由于采用迭代器, 每个收到的结果均能马上显示出来, 一旦捕获到的minions的结果大于等于之前获得的minions数目, 即表示所有minions均已返回结果, 退出.</li>
</ol>
</div>
</div>
<div class="section" id="section-4">
<h2>总结</h2>
<p>Salt利用ZeroMQ灵活高效的patterns, 使Salt网络拓扑变得非常灵活高效. 利用PUB/SUB, 实现了高效的远程执行指令下发机制; 利用ROUTER/REQ, 实现认证及异步的远程执行结果返回; 利用DEALER/REP, 实现多进程任务处理机制; 利用PULL/PUB, 实现Event BUS, 使其他或第三方应用可以快速的使用PUB/SUB接收到Event BUS上的消息.</p>
<p>I love Salt, I love ZeroMQ!</p>
</div>
Salt中Pillar那点事2014-06-08T00:00:00+08:002014-06-08T00:00:00+08:00pengyaotag:pengyao.org,2014-06-08:/salt-pillar-01.html<p class="first last">在 <a class="reference external" href="http://saltstack.com/">SaltStack</a> 中, Pillar作为定义minion全局数据的接口. 那么在Salt内部, Pillar是如何工作的? 在哪些情况下, 使用Pillar需要先执行刷新操作? 而哪些又不需要?</p>
<div class="section" id="section-1">
<h2>基本简介</h2>
<p>在 <a class="reference external" href="http://saltstack.com/">SaltStack</a> 中, Pillar作为定义minion全局数据的接口. 默认存储在master端, Minion启动时会连接master获取最新的pillar数据. Pillar使用类似于State Tree的结构, 默认使用 <cite>YAML</cite> 作为其描述格式, 在Minion内部最终转换成 <a class="reference external" href="https://docs.python.org/2/tutorial/datastructures.html#dictionaries">Python字典</a> .</p>
<p>那么在Salt内部, Pillar是如何工作的? 在哪些情况下,需要先执行刷新Pillar操作? 而哪些又不需要?</p>
<p>本文基于 <a class="reference external" href="http://docs.saltstack.com/en/latest/topics/releases/2014.1.4.html">Salt 2014.1.4</a></p>
</div>
<div class="section" id="pillar">
<h2>配置文件中的Pillar</h2>
<dl class="docutils">
<dt>pillar_roots</dt>
<dd>存在于master/minion配置文件中. 指定Pillar roots对应环境的目录, 其布局类似于State Tree. 在minion配置文件中配置该选项, 只有当 <em>file_client</em> 为 <em>local</em> 时才生效.</dd>
<dt>state_top</dt>
<dd>存在于master/minion配置文件中, 默认值为top.sls. 官方描述为用于state system, 用于告诉minion使用哪个环境并且需要执行哪些模块. 其实该选项也应用在pillar system中, 作用和state system类似. 所以如果更改了本选项, pillar system对应的top.sls也需要变更. 在minion配置文件中配置该选项, 只有当 <em>file_client</em> 为 <em>local</em> 时才生效.</dd>
<dt>file_client</dt>
<dd>存在于minion配置文件中, 默认值为remote. 用于指定去哪里查找文件. 有效值是 <em>remote</em> 和 <em>local</em>. <em>remote</em> 表示使用master, <em>local</em> 用于 <a class="reference external" href="http://docs.saltstack.com/en/latest/topics/tutorials/quickstart.html#telling-salt-to-run-masterless">Masterless</a> 的情况.</dd>
<dt>pillar_opts</dt>
<dd>存在于master配置文件中, 默认值为True. 指定是否将master配置选项作为pillar. 如果该选项为True, 修改了master配置选项时, 需要重启master, 才能在pillar中得到最新的值.</dd>
</dl>
</div>
<div class="section" id="minionpillar">
<h2>Minion中的Pillar实现</h2>
<p>Minion中pillar为Python字典, Minion启动时, 默认会连接master获取最新的pillar数据, 存储在 <em>self.opts['pillar']</em> 中. <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/minion.py#L520">对应代码</a> 如下:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Minion</span><span class="p">(</span><span class="n">MinionBase</span><span class="p">):</span>
<span class="w"> </span><span class="sd">'''</span>
<span class="sd"> This class instantiates a minion, runs connections for a minion,</span>
<span class="sd"> and loads all of the functions into the minion</span>
<span class="sd"> '''</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">opts</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">60</span><span class="p">,</span> <span class="n">safe</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
<span class="w"> </span><span class="sd">'''</span>
<span class="sd"> Pass in the options dict</span>
<span class="sd"> '''</span>
<span class="o">......</span>
<span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">[</span><span class="s1">'pillar'</span><span class="p">]</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">pillar</span><span class="o">.</span><span class="n">get_pillar</span><span class="p">(</span>
<span class="n">opts</span><span class="p">,</span>
<span class="n">opts</span><span class="p">[</span><span class="s1">'grains'</span><span class="p">],</span>
<span class="n">opts</span><span class="p">[</span><span class="s1">'id'</span><span class="p">],</span>
<span class="n">opts</span><span class="p">[</span><span class="s1">'environment'</span><span class="p">],</span>
<span class="p">)</span><span class="o">.</span><span class="n">compile_pillar</span><span class="p">()</span>
<span class="o">......</span>
</pre></div>
<p>那么 <em>salt.pillar.get_pillar</em> 是如何工作的? <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/pillar/__init__.py#L28">对应代码</a> 如下:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">get_pillar</span><span class="p">(</span><span class="n">opts</span><span class="p">,</span> <span class="n">grains</span><span class="p">,</span> <span class="n">id_</span><span class="p">,</span> <span class="n">saltenv</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">ext</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">env</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="w"> </span><span class="sd">'''</span>
<span class="sd"> Return the correct pillar driver based on the file_client option</span>
<span class="sd"> '''</span>
<span class="k">if</span> <span class="n">env</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="n">salt</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">warn_until</span><span class="p">(</span>
<span class="s1">'Boron'</span><span class="p">,</span>
<span class="s1">'Passing a salt environment should be done using </span><span class="se">\'</span><span class="s1">saltenv</span><span class="se">\'</span><span class="s1"> '</span>
<span class="s1">'not </span><span class="se">\'</span><span class="s1">env</span><span class="se">\'</span><span class="s1">. This functionality will be removed in Salt Boron.'</span>
<span class="p">)</span>
<span class="c1"># Backwards compatibility</span>
<span class="n">saltenv</span> <span class="o">=</span> <span class="n">env</span>
<span class="k">return</span> <span class="p">{</span>
<span class="s1">'remote'</span><span class="p">:</span> <span class="n">RemotePillar</span><span class="p">,</span>
<span class="s1">'local'</span><span class="p">:</span> <span class="n">Pillar</span>
<span class="p">}</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">opts</span><span class="p">[</span><span class="s1">'file_client'</span><span class="p">],</span> <span class="n">Pillar</span><span class="p">)(</span><span class="n">opts</span><span class="p">,</span> <span class="n">grains</span><span class="p">,</span> <span class="n">id_</span><span class="p">,</span> <span class="n">saltenv</span><span class="p">,</span> <span class="n">ext</span><span class="p">)</span>
</pre></div>
<p>也可以从代码中获知, 会从opts中获取 <em>file_client</em> 值, 如果是remote, 则对应的对象为RemotePillar, 如果是local, 则为Pillar, 进行后续处理</p>
<p>如果Minion在运行过程中, 接受到的指令以 <em>refresh_pillar</em> 字符串开头, 则执行 <em>pillar_refresh</em> 操作. <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/minion.py#L1376">对应代码</a> 如下:</p>
<div class="highlight"><pre><span></span><span class="k">if</span> <span class="n">package</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'module_refresh'</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">module_refresh</span><span class="p">()</span>
<span class="k">elif</span> <span class="n">package</span><span class="o">.</span><span class="n">startswith</span><span class="p">(</span><span class="s1">'pillar_refresh'</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">pillar_refresh</span><span class="p">()</span>
</pre></div>
<p>那么 <em>pillar_refresh()</em> 都进行了哪些工作? <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/minion.py#L1090">对应代码</a> 如下:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">pillar_refresh</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">'''</span>
<span class="sd"> Refresh the pillar</span>
<span class="sd"> '''</span>
<span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">[</span><span class="s1">'pillar'</span><span class="p">]</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">pillar</span><span class="o">.</span><span class="n">get_pillar</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">[</span><span class="s1">'grains'</span><span class="p">],</span>
<span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">[</span><span class="s1">'id'</span><span class="p">],</span>
<span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">[</span><span class="s1">'environment'</span><span class="p">],</span>
<span class="p">)</span><span class="o">.</span><span class="n">compile_pillar</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">module_refresh</span><span class="p">()</span>
</pre></div>
<p>从代码中得知, pillar_refresh操作, 除了从Master端/Minion本地获取最新的pillar信息外, 也会执行模块刷新(module_refresh)工作. 可以将minion本地的日志级别调整为 <em>trac</em>, 然后执行 <em>saltutil.refresh_pillar</em> 操作, 然后观察minion日志, 是否会刷新模块进行验证.</p>
</div>
<div class="section" id="targetpillar">
<h2>Target中的Pillar</h2>
<p>Salt指令发送底层网络, 采用ZeroMQ PUB/SUB结构. Minion会监听SUB接口, Master会将指令发送到本地的PUB接口, 然后所有Minion均会收到该指令, 然后在Minion本地判断自己是否需要执行该指令(即Target). 当前版本中, 已经支持pillar作为Target(通过"-I"选项指定). <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/minion.py#L1809">对应代码</a> 如下:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">pillar_match</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">tgt</span><span class="p">,</span> <span class="n">delim</span><span class="o">=</span><span class="s1">':'</span><span class="p">):</span>
<span class="w"> </span><span class="sd">'''</span>
<span class="sd"> Reads in the pillar glob match</span>
<span class="sd"> '''</span>
<span class="n">log</span><span class="o">.</span><span class="n">debug</span><span class="p">(</span><span class="s1">'pillar target: </span><span class="si">{0}</span><span class="s1">'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="n">tgt</span><span class="p">))</span>
<span class="k">if</span> <span class="n">delim</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">tgt</span><span class="p">:</span>
<span class="n">log</span><span class="o">.</span><span class="n">error</span><span class="p">(</span><span class="s1">'Got insufficient arguments for pillar match '</span>
<span class="s1">'statement from master'</span><span class="p">)</span>
<span class="k">return</span> <span class="kc">False</span>
<span class="k">return</span> <span class="n">salt</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">subdict_match</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">[</span><span class="s1">'pillar'</span><span class="p">],</span> <span class="n">tgt</span><span class="p">,</span> <span class="n">delim</span><span class="o">=</span><span class="n">delim</span><span class="p">)</span>
</pre></div>
<p>可以看出, 其匹配使用的是 <em>self.opts['pillar']</em> 即当前Minion内存中的Pillar的数据. 因此如果在Master/Minion(当 <em>file_client</em> 为 <em>local</em> 时)修改了Pillar数据后, 想要使用最新的Pillar来做Target操作, 需要在执行前先手动执行 <em>saltutil.refresh_pillar</em> 操作, 以刷新Minion内存中的Pillar数据.</p>
</div>
<div class="section" id="pillar-1">
<h2>远程执行模块中的Pillar</h2>
<div class="section" id="pillar-items">
<h3>pillar.items</h3>
<p><a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/modules/pillar.py#L42">对应代码</a> 如下:</p>
<div class="highlight"><pre><span></span><span class="n">pillar</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">pillar</span><span class="o">.</span><span class="n">get_pillar</span><span class="p">(</span>
<span class="n">__opts__</span><span class="p">,</span>
<span class="n">__grains__</span><span class="p">,</span>
<span class="n">__opts__</span><span class="p">[</span><span class="s1">'id'</span><span class="p">],</span>
<span class="n">__opts__</span><span class="p">[</span><span class="s1">'environment'</span><span class="p">])</span>
<span class="k">return</span> <span class="n">pillar</span><span class="o">.</span><span class="n">compile_pillar</span><span class="p">()</span>
</pre></div>
<p>会连接Master/Minion(当 <em>file_client</em> 为 <em>local</em> 时)获取最新的pillar数据并返回. 但并不会刷新Minion本地的缓存. 也就是说, 在master端修改了Pillar Tree, 在刷新pillar(saltutil.refresh_pillar)前, 可以先使用 <em>pillar.items</em> 来验证其数据是否达到预期.</p>
</div>
<div class="section" id="pillar-data">
<h3>pillar.data</h3>
<p><a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/modules/pillar.py#L67">对应代码</a> 如下:</p>
<div class="highlight"><pre><span></span><span class="n">data</span> <span class="o">=</span> <span class="n">items</span>
</pre></div>
<p>只是创建了一个赋值引用, 指定data和执行items一样</p>
</div>
<div class="section" id="pillar-item">
<h3>pillar.item</h3>
<p><a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/modules/pillar.py#L70">对应代码</a> 如下:</p>
<div class="highlight"><pre><span></span><span class="n">ret</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">pillar</span> <span class="o">=</span> <span class="n">items</span><span class="p">()</span>
<span class="k">for</span> <span class="n">arg</span> <span class="ow">in</span> <span class="n">args</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">ret</span><span class="p">[</span><span class="n">arg</span><span class="p">]</span> <span class="o">=</span> <span class="n">pillar</span><span class="p">[</span><span class="n">arg</span><span class="p">]</span>
<span class="k">except</span> <span class="ne">KeyError</span><span class="p">:</span>
<span class="k">pass</span>
<span class="k">return</span> <span class="n">ret</span>
</pre></div>
<p>先使用pillar.items来获取最新的Master端最新的pillar数据. 然后一个for循环, 从items获取所需要的keys对应的值. 所以item可以查询多个key.</p>
</div>
<div class="section" id="pillar-raw">
<h3>pillar.raw</h3>
<p><a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/modules/pillar.py#L93">对应代码</a> 如下:</p>
<div class="highlight"><pre><span></span><span class="k">if</span> <span class="n">key</span><span class="p">:</span>
<span class="n">ret</span> <span class="o">=</span> <span class="n">__pillar__</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">,</span> <span class="p">{})</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">ret</span> <span class="o">=</span> <span class="n">__pillar__</span>
<span class="k">return</span> <span class="n">ret</span>
</pre></div>
<p>从当前Minion本地获取 __pillar__ (self.opts[pillar])的值. 也就是说使用 <em>pillar.raw</em> 与 <em>pillar.items</em> 不同, 获取到的是Minion内存中的pillar的值, 并非是master端定义的值. 如果指定了key, 则返回对应key的值. 如果没有, 则返回整个 __pillar__</p>
</div>
<div class="section" id="pillar-get">
<h3>pillar.get</h3>
<p><a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/modules/pillar.py#L16">对应代码</a> 如下:</p>
<div class="highlight"><pre><span></span><span class="k">return</span> <span class="n">salt</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">traverse_dict</span><span class="p">(</span><span class="n">__pillar__</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">default</span><span class="p">)</span>
</pre></div>
<p>和 <em>pillar.raw</em> 工作方式类似, 是从 __pillar__ 中进行的取值, 用于获取pillar中对应的key值. 与 pillar.raw执行key不同的是, get递归获取内嵌字典的值(默认以":"做分隔). 从最新develop分支中看, 下一个版本(Helium)中将增加merge功能.</p>
</div>
<div class="section" id="pillar-ext">
<h3>pillar.ext</h3>
<p>与pillar.items工作方式类似, 用于获取ext pillar的值</p>
</div>
<div class="section" id="saltutil-refresh-pillar">
<h3>saltutil.refresh_pillar</h3>
<p><a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/modules/saltutil.py#L335">对应代码</a> 如下:</p>
<div class="highlight"><pre><span></span><span class="n">__salt__</span><span class="p">[</span><span class="s1">'event.fire'</span><span class="p">]({},</span> <span class="s1">'pillar_refresh'</span><span class="p">)</span>
</pre></div>
<p>在Minion本地Event接口上产生一个 <em>pillar_refresh</em> event. 之前在Minion中的Pillar中, Minion本地会监听本地Event接口, 如果捕捉到以 <em>pillar_refresh</em> 开始的指令, 会刷新本地pillar.</p>
</div>
</div>
<div class="section" id="pillar-2">
<h2>配置管理中的Pillar</h2>
<div class="section" id="slspillar">
<h3>在SLS中使用Pillar</h3>
<p>在SLS中, 可以直接使用pillar. 如pillar['pkg'], 其直接使用的是Minion当前内存中pillar的值(self.opts['pillar']).</p>
</div>
<div class="section" id="state-sls-state-highstate">
<h3>state.sls & state.highstate</h3>
<p>将这两个远程执行模块方法放到配置管理中, 因为其功能是用于向Minions发送配置管理指令.</p>
<p>state.sls及state.highstate在代码中, 均为 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/state.py#L2574">salt.state.HighState</a> 对象. 在执行时为 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/state.py#L526">State</a> 对象. State类在实例化时,则会刷新pillar, <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/state.py#L530">对应代码</a> 如下:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">State</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="w"> </span><span class="sd">'''</span>
<span class="sd"> Class used to execute salt states</span>
<span class="sd"> '''</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">opts</span><span class="p">,</span> <span class="n">pillar</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">jid</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
<span class="k">if</span> <span class="s1">'grains'</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">opts</span><span class="p">:</span>
<span class="n">opts</span><span class="p">[</span><span class="s1">'grains'</span><span class="p">]</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">loader</span><span class="o">.</span><span class="n">grains</span><span class="p">(</span><span class="n">opts</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">opts</span> <span class="o">=</span> <span class="n">opts</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_pillar_override</span> <span class="o">=</span> <span class="n">pillar</span>
<span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">[</span><span class="s1">'pillar'</span><span class="p">]</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">_gather_pillar</span><span class="p">()</span>
</pre></div>
<p>而_gather_pillar <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/state.py#L544">对应代码</a> 如下:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">_gather_pillar</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="w"> </span><span class="sd">'''</span>
<span class="sd"> Whenever a state run starts, gather the pillar data fresh</span>
<span class="sd"> '''</span>
<span class="n">pillar</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">pillar</span><span class="o">.</span><span class="n">get_pillar</span><span class="p">(</span>
<span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">,</span>
<span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">[</span><span class="s1">'grains'</span><span class="p">],</span>
<span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">[</span><span class="s1">'id'</span><span class="p">],</span>
<span class="bp">self</span><span class="o">.</span><span class="n">opts</span><span class="p">[</span><span class="s1">'environment'</span><span class="p">],</span>
<span class="p">)</span>
<span class="n">ret</span> <span class="o">=</span> <span class="n">pillar</span><span class="o">.</span><span class="n">compile_pillar</span><span class="p">()</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_pillar_override</span> <span class="ow">and</span> <span class="nb">isinstance</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_pillar_override</span><span class="p">,</span> <span class="nb">dict</span><span class="p">):</span>
<span class="n">ret</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">_pillar_override</span><span class="p">)</span>
<span class="k">return</span> <span class="n">ret</span>
</pre></div>
<p>_gather_pillar从Master上获取Minion对应的最新pillar数据, __init__方法中的 <em>self.opts['pillar'] = self._gather_pillar()</em> 将该数据赋值给self.opts['pillar']以完成Minion本地内存中Pillar数据的刷新操作. 这就是为什么修改了Master上的Pillar的值, 而无需执行刷新操作(saltutil.refresh_pillar), 因为在执行state.highstate及state.sls时会自动应该最新的值.</p>
</div>
</div>
<div class="section" id="ext-pillar">
<h2>ext_pillar</h2>
<p>Salt支持从第三方系统中获取Pillar信息,使Salt易于与现有的CMDB系统进行数据整合. 对应的配置是master配置文件中的ext_pillar选项. 官方当前已经提供了 <a class="reference external" href="http://docs.saltstack.com/en/latest/ref/pillar/all/">若干驱动</a> .</p>
<p>如果已经提供的驱动并不满足需求, 自定义ext_pillar驱动也非常简单. 只需要驱动文件放到master端salt代码中pillar目录下即可, 驱动为python代码, 其中包含ext_pillar函数, 且该函数第一个参数是minion_id, 第二个参数为pillar, 其返回值是一个标准的 <a class="reference external" href="https://docs.python.org/2/tutorial/datastructures.html#dictionaries">Python字典</a> 即可. 可以参照 <a class="reference external" href="https://github.com/saltstack/salt/blob/v2014.1.4/salt/pillar/cobbler.py">cobbler的ext_pillar</a> 进行编写.</p>
</div>
基于Salt Event系统构建Master端returner2014-04-18T00:00:00+08:002014-04-18T00:00:00+08:00pengyaotag:pengyao.org,2014-04-18:/saltstack_master_retuner_over_event_system.html<p class="first last">SaltStack的returner是由minion端主动连接returner完成执行结果的存储, 在部分场景下并不能满足需求. 由于Salt底层已经构建了一套Event系统, 所有的操作均会产生event. 因此基于Salt Event System构建Master端returner成为一种可能.</p>
<p><a class="reference external" href="http://saltstack.com/">SaltStack</a> 的 <a class="reference external" href="http://docs.saltstack.com/en/latest/ref/returners/">returner</a> 是由minion端主动连接returner完成执行结果的存储, 在部分场景下并不能满足需求. 由于Salt底层已经构建了一套 <a class="reference external" href="http://docs.saltstack.com/en/latest/topics/event/index.html">Event系统</a> , 所有的操作均会产生event. 因此基于Salt Event系统构建Master端returner成为一种可能.</p>
<p>之前已经完成了 <a class="reference external" href="http://pengyao.org/saltstack_event_system_listen_events.html">SaltStack Event系统监听events测试</a>, 本文将基于Salt Event系统构建Master端returner.</p>
<div class="section" id="section-1">
<h2>前置阅读</h2>
<ul class="simple">
<li>SaltStack Event系统: <a class="reference external" href="http://docs.saltstack.com/en/latest/topics/event/index.html">http://docs.saltstack.com/en/latest/topics/event/index.html</a></li>
<li>SaltStack Event系统监听events测试: <a class="reference external" href="http://pengyao.org/saltstack_event_system_listen_events.html">http://pengyao.org/saltstack_event_system_listen_events.html</a></li>
</ul>
</div>
<div class="section" id="section-2">
<h2>环境说明</h2>
<ul class="simple">
<li>测试结构: Master/Minions结构, 共一台minion, 对应id为: <em>salt-minion-01.example.com</em></li>
<li>Salt Version: <em>2014.1.1</em></li>
<li>本次测试结果将存放在MySQL中, 为了方便测试, 已经在Master本地部署了MySQL Server</li>
</ul>
</div>
<div class="section" id="section-3">
<h2>开工</h2>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">以下操作如非特别注明, 均在Master端进行</p>
</div>
<div class="section" id="section-4">
<h3>前置配置</h3>
<p>安装MySQLdb依赖</p>
<div class="highlight"><pre><span></span>yum<span class="w"> </span>-y<span class="w"> </span>install<span class="w"> </span>MySQL-python
</pre></div>
<p>配置本次测试需要使用的数据库及用户</p>
<div class="highlight"><pre><span></span><span class="c1"># 创建salt数据库</span>
mysql<span class="w"> </span>-e<span class="w"> </span><span class="s1">'create database salt'</span>
<span class="c1"># 创建用于连接salt数据库的用户</span>
mysql<span class="w"> </span>-e<span class="w"> </span><span class="s1">'"grant all on salt.* to salt@localhost identified by "salt_pass'</span><span class="p">;</span>
<span class="c1"># 将数据库配置添加至master配置文件中</span>
<span class="nb">echo</span><span class="w"> </span>-e<span class="w"> </span><span class="s2">"\n\n# MySQL\nmysql.host: 'localhost'\nmysql.user: 'salt'\nmysql.pass: 'salt_pass'\nmysql.db: 'salt'\nmysql.port: 3306"</span><span class="w"> </span>>><span class="w"> </span>/etc/salt/master
</pre></div>
<p>为了与salt自带的 <a class="reference external" href="http://docs.saltstack.com/en/latest/ref/returners/all/salt.returners.mysql.html">mysql returner</a> 兼容, 本次直接使用mysql retuner对应的数据库表结构:</p>
<pre class="literal-block">
USE `salt`;
--
-- Table structure for table `jids`
--
DROP TABLE IF EXISTS `jids`;
CREATE TABLE `jids` (
`jid` varchar(255) NOT NULL,
`load` mediumtext NOT NULL,
UNIQUE KEY `jid` (`jid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
--
-- Table structure for table `salt_returns`
--
DROP TABLE IF EXISTS `salt_returns`;
CREATE TABLE `salt_returns` (
`fun` varchar(50) NOT NULL,
`jid` varchar(255) NOT NULL,
`return` mediumtext NOT NULL,
`id` varchar(255) NOT NULL,
`success` varchar(10) NOT NULL,
`full_ret` mediumtext NOT NULL,
`alter_time` TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
KEY `id` (`id`),
KEY `jid` (`jid`),
KEY `fun` (`fun`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
</pre>
</div>
<div class="section" id="returner">
<h3>编写returner</h3>
<p><em>salt_event_to_mysql.py</em></p>
<div class="highlight"><pre><span></span><span class="ch">#!/bin/env python</span>
<span class="c1">#coding=utf8</span>
<span class="c1"># Import python libs</span>
<span class="kn">import</span> <span class="nn">json</span>
<span class="c1"># Import salt modules</span>
<span class="kn">import</span> <span class="nn">salt.config</span>
<span class="kn">import</span> <span class="nn">salt.utils.event</span>
<span class="c1"># Import third party libs</span>
<span class="kn">import</span> <span class="nn">MySQLdb</span>
<span class="n">__opts__</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">config</span><span class="o">.</span><span class="n">client_config</span><span class="p">(</span><span class="s1">'/etc/salt/master'</span><span class="p">)</span>
<span class="c1"># Create MySQL connect</span>
<span class="n">conn</span> <span class="o">=</span> <span class="n">MySQLdb</span><span class="o">.</span><span class="n">connect</span><span class="p">(</span><span class="n">host</span><span class="o">=</span><span class="n">__opts__</span><span class="p">[</span><span class="s1">'mysql.host'</span><span class="p">],</span> <span class="n">user</span><span class="o">=</span><span class="n">__opts__</span><span class="p">[</span><span class="s1">'mysql.user'</span><span class="p">],</span> <span class="n">passwd</span><span class="o">=</span><span class="n">__opts__</span><span class="p">[</span><span class="s1">'mysql.pass'</span><span class="p">],</span> <span class="n">db</span><span class="o">=</span><span class="n">__opts__</span><span class="p">[</span><span class="s1">'mysql.db'</span><span class="p">],</span> <span class="n">port</span><span class="o">=</span><span class="n">__opts__</span><span class="p">[</span><span class="s1">'mysql.port'</span><span class="p">])</span>
<span class="n">cursor</span> <span class="o">=</span> <span class="n">conn</span><span class="o">.</span><span class="n">cursor</span><span class="p">()</span>
<span class="c1"># Listen Salt Master Event System</span>
<span class="n">event</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">event</span><span class="o">.</span><span class="n">MasterEvent</span><span class="p">(</span><span class="n">__opts__</span><span class="p">[</span><span class="s1">'sock_dir'</span><span class="p">])</span>
<span class="k">for</span> <span class="n">eachevent</span> <span class="ow">in</span> <span class="n">event</span><span class="o">.</span><span class="n">iter_events</span><span class="p">(</span><span class="n">full</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
<span class="n">ret</span> <span class="o">=</span> <span class="n">eachevent</span><span class="p">[</span><span class="s1">'data'</span><span class="p">]</span>
<span class="k">if</span> <span class="s2">"salt/job/"</span> <span class="ow">in</span> <span class="n">eachevent</span><span class="p">[</span><span class="s1">'tag'</span><span class="p">]:</span>
<span class="c1"># Return Event</span>
<span class="k">if</span> <span class="n">ret</span><span class="o">.</span><span class="n">has_key</span><span class="p">(</span><span class="s1">'id'</span><span class="p">)</span> <span class="ow">and</span> <span class="n">ret</span><span class="o">.</span><span class="n">has_key</span><span class="p">(</span><span class="s1">'return'</span><span class="p">):</span>
<span class="c1"># Igonre saltutil.find_job event</span>
<span class="k">if</span> <span class="n">ret</span><span class="p">[</span><span class="s1">'fun'</span><span class="p">]</span> <span class="o">==</span> <span class="s2">"saltutil.find_job"</span><span class="p">:</span>
<span class="k">continue</span>
<span class="n">sql</span> <span class="o">=</span> <span class="s1">'''INSERT INTO `salt_returns`</span>
<span class="s1"> (`fun`, `jid`, `return`, `id`, `success`, `full_ret` )</span>
<span class="s1"> VALUES (</span><span class="si">%s</span><span class="s1">, </span><span class="si">%s</span><span class="s1">, </span><span class="si">%s</span><span class="s1">, </span><span class="si">%s</span><span class="s1">, </span><span class="si">%s</span><span class="s1">, </span><span class="si">%s</span><span class="s1">)'''</span>
<span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="n">sql</span><span class="p">,</span> <span class="p">(</span><span class="n">ret</span><span class="p">[</span><span class="s1">'fun'</span><span class="p">],</span> <span class="n">ret</span><span class="p">[</span><span class="s1">'jid'</span><span class="p">],</span>
<span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">ret</span><span class="p">[</span><span class="s1">'return'</span><span class="p">]),</span> <span class="n">ret</span><span class="p">[</span><span class="s1">'id'</span><span class="p">],</span>
<span class="n">ret</span><span class="p">[</span><span class="s1">'success'</span><span class="p">],</span> <span class="n">json</span><span class="o">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">ret</span><span class="p">)))</span>
<span class="n">cursor</span><span class="o">.</span><span class="n">execute</span><span class="p">(</span><span class="s2">"COMMIT"</span><span class="p">)</span>
<span class="c1"># Other Event</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">pass</span>
</pre></div>
<p>运行本returner:</p>
<div class="highlight"><pre><span></span>python<span class="w"> </span>salt_event_to_mysql.py
</pre></div>
</div>
<div class="section" id="section-5">
<h3>测试</h3>
<p>新开启一个终端, 运行Salt指令:</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>test.ping
</pre></div>
<p>输出为:</p>
<div class="highlight"><pre><span></span><span class="nt">salt-minion-01.example.com</span><span class="p">:</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">True</span>
</pre></div>
<p>检查mysql数据库, 查询salt_returns表数据:</p>
<div class="highlight"><pre><span></span>mysql<span class="w"> </span>salt<span class="w"> </span>-e<span class="w"> </span><span class="s2">"select * from salt_returns\G"</span>
</pre></div>
<p>输出为:</p>
<pre class="literal-block">
*************************** 1. row ***************************
fun: test.ping
jid: 20140417161103569310
return: true
id: salt-minion-01.example.com
success: 1
full_ret: {"fun_args": [], "jid": "20140417161103569310", "return": true, "retcode": 0, "success": true, "cmd": "_return", "_stamp": "2014-04-17T16:11:03.584859", "fun": "test.ping", "id": "salt-minion-01.example.com"}
alter_time: 2014-04-17 16:11:03
</pre>
<p>入库成功</p>
</div>
</div>
SaltStack Event系统监听events测试2014-04-17T00:00:00+08:002014-04-17T00:00:00+08:00pengyaotag:pengyao.org,2014-04-17:/saltstack_event_system_listen_events.html<p class="first last">SaltStack 0.10版本中, 基于ZeroMQ publish socket, 新增了Event系统, 允许其他进程连接该socket并监听event bus上的events. 本文对SaltStack Event系统进行监听events测试.</p>
<p>SaltStack 0.10版本中, 新增了Event系统, 官方在 <a class="reference external" href="http://docs.saltstack.com/en/latest/topics/releases/0.10.0.html#event-system">Release Notes</a> 对其描述如下:</p>
<pre class="literal-block">
The Salt Master now comes equipped with a new event system. This event system has replaced some of the back end of the Salt client and offers the beginning of a system which will make plugging external applications into Salt. The event system relies on a local ZeroMQ publish socket and other processes can connect to this socket and listen for events. The new events can be easily managed via Salt's event library.
</pre>
<p>同时官方也在 <a class="reference external" href="http://docs.saltstack.com/en/latest/topics/event/index.html#listening-for-events">Salt Event系统</a> 页面中提供了监听event的例子程序, 基于其进行下Event系统学习.</p>
<div class="section" id="section-1">
<h2>环境说明</h2>
<ul class="simple">
<li>测试结构: Master/Minions结构, 共一台minion, 对应id为: <em>salt-minion-01.example.com</em></li>
<li>Salt Version: <em>2014.1.1</em></li>
</ul>
</div>
<div class="section" id="section-2">
<h2>开工</h2>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">Salt Event系统页面中, 官方强调Event系统可以通过event library访问, 并且只允许和Salt运行的用户访问. 由于运行master时是通过root用户运行, 所以以下操作均在root用户下进行. 如非特别注明, 均在Master端进行操作.</p>
</div>
<p>新开一个终端, 运行python, 基于其尝试监听所有的Event:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">salt.utils.event</span>
<span class="n">event</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">utils</span><span class="o">.</span><span class="n">event</span><span class="o">.</span><span class="n">MasterEvent</span><span class="p">(</span><span class="s1">'/var/run/salt/master'</span><span class="p">)</span>
<span class="k">for</span> <span class="n">eachevent</span> <span class="ow">in</span> <span class="n">event</span><span class="o">.</span><span class="n">iter_events</span><span class="p">(</span><span class="n">full</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
<span class="nb">print</span> <span class="n">eachevent</span>
<span class="nb">print</span> <span class="s2">"------"</span>
</pre></div>
<p>在另外一个终端执行:</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>test.ping
</pre></div>
<p>查看之前监听所有Event的终端, 有如下输出:</p>
<pre class="literal-block">
{'tag': '20140417135823133764', 'data': {'_stamp': '2014-04-17T13:58:23.133956', 'minions': ['salt-minion-01.example.com']}}
------
{'tag': 'new_job', 'data': {'tgt_type': 'glob', 'jid': '20140417135823133764', 'tgt': '*', '_stamp': '2014-04-17T13:58:23.134005', 'user': 'sudo_vagrant', 'arg': [], 'fun': 'test.ping', 'minions': ['salt-minion-01.example.com']}}
------
{'tag': 'salt/job/20140417135823133764/new', 'data': {'tgt_type': 'glob', 'jid': '20140417135823133764', 'tgt': '*', '_stamp': '2014-04-17T13:58:23.134064', 'user': 'sudo_vagrant', 'arg': [], 'fun': 'test.ping', 'minions': ['salt-minion-01.example.com']}}
------
{'tag': '20140417135823133764', 'data': {'fun_args': [], 'jid': '20140417135823133764', 'return': True, 'retcode': 0, 'success': True, 'cmd': '_return', '_stamp': '2014-04-17T13:58:23.150356', 'fun': 'test.ping', 'id': 'salt-minion-01.example.com'}}
------
{'tag': 'salt/job/20140417135823133764/ret/salt-minion-01.example.com', 'data': {'fun_args': [], 'jid': '20140417135823133764', 'return': True, 'retcode': 0, 'success': True, 'cmd': '_return', '_stamp': '2014-04-17T13:58:23.150397', 'fun': 'test.ping', 'id': 'salt-minion-01.example.com'}}
------
</pre>
<p>从输出结果看, 对于tag只是jid的, 官方在源码中标记的注释是"old dup event", 推测是为了兼容旧的event系统(0.17.0版本event系统进行了更新), 本次对其不做处理. 下发任务对应的tag为 <em>new_job</em>, 并且下发任务时就master端就在event中注定了那些minions需要运行(对应的data字典中的minions). 如果tag中包含 <em>salt/job/</em> 字样并且data字典中 <em>return</em> 为True, 则表示该Event是minion返回的结果.</p>
<p>同时测试下超过timeout设置(默认为5秒)的任务:</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>cmd.run<span class="w"> </span><span class="s1">'sleep 6; echo hello world'</span>
</pre></div>
<p>输出结果为:</p>
<pre class="literal-block">
{'tag': '20140417141834578593', 'data': {'_stamp': '2014-04-17T14:18:34.578822', 'minions': ['salt-minion-01.example.com']}}
------
{'tag': 'new_job', 'data': {'tgt_type': 'glob', 'jid': '20140417141834578593', 'tgt': '*', '_stamp': '2014-04-17T14:18:34.578881', 'user': 'sudo_vagrant', 'arg': ['sleep 6; echo hello world'], 'fun': 'cmd.run', 'minions': ['salt-minion-01.example.com']}}
------
{'tag': 'salt/job/20140417141834578593/new', 'data': {'tgt_type': 'glob', 'jid': '20140417141834578593', 'tgt': '*', '_stamp': '2014-04-17T14:18:34.578917', 'user': 'sudo_vagrant', 'arg': ['sleep 6; echo hello world'], 'fun': 'cmd.run', 'minions': ['salt-minion-01.example.com']}}
------
{'tag': '20140417141839587706', 'data': {'_stamp': '2014-04-17T14:18:39.587908', 'minions': ['salt-minion-01.example.com']}}
------
{'tag': 'new_job', 'data': {'tgt_type': 'glob', 'jid': '20140417141839587706', 'tgt': '*', '_stamp': '2014-04-17T14:18:39.587961', 'user': 'sudo_vagrant', 'arg': ['20140417141834578593'], 'fun': 'saltutil.find_job', 'minions': ['salt-minion-01.example.com']}}
------
{'tag': 'salt/job/20140417141839587706/new', 'data': {'tgt_type': 'glob', 'jid': '20140417141839587706', 'tgt': '*', '_stamp': '2014-04-17T14:18:39.587985', 'user': 'sudo_vagrant', 'arg': ['20140417141834578593'], 'fun': 'saltutil.find_job', 'minions': ['salt-minion-01.example.com']}}
------
{'tag': '20140417141839587706', 'data': {'fun_args': ['20140417141834578593'], 'jid': '20140417141839587706', 'return': {'tgt_type': 'glob', 'jid': '20140417141834578593', 'tgt': '*', 'pid': 2143, 'ret': '', 'user': 'sudo_vagrant', 'arg': ['sleep 6; echo hello world'], 'fun': 'cmd.run'}, 'retcode': 0, 'success': True, 'cmd': '_return', '_stamp': '2014-04-17T14:18:39.605262', 'fun': 'saltutil.find_job', 'id': 'salt-minion-01.example.com'}}
------
{'tag': 'salt/job/20140417141839587706/ret/salt-minion-01.example.com', 'data': {'fun_args': ['20140417141834578593'], 'jid': '20140417141839587706', 'return': {'tgt_type': 'glob', 'jid': '20140417141834578593', 'tgt': '*', 'pid': 2143, 'ret': '', 'user': 'sudo_vagrant', 'arg': ['sleep 6; echo hello world'], 'fun': 'cmd.run'}, 'retcode': 0, 'success': True, 'cmd': '_return', '_stamp': '2014-04-17T14:18:39.605321', 'fun': 'saltutil.find_job', 'id': 'salt-minion-01.example.com'}}
------
{'tag': '20140417141834578593', 'data': {'fun_args': ['sleep 6; echo hello world'], 'jid': '20140417141834578593', 'return': 'hello world', 'retcode': 0, 'success': True, 'cmd': '_return', '_stamp': '2014-04-17T14:18:40.604562', 'fun': 'cmd.run', 'id': 'salt-minion-01.example.com'}}
------
{'tag': 'salt/job/20140417141834578593/ret/salt-minion-01.example.com', 'data': {'fun_args': ['sleep 6; echo hello world'], 'jid': '20140417141834578593', 'return': 'hello world', 'retcode': 0, 'success': True, 'cmd': '_return', '_stamp': '2014-04-17T14:18:40.604628', 'fun': 'cmd.run', 'id': 'salt-minion-01.example.com'}}
------
</pre>
<p>除了之前test.ping测试类似的输出外, 可以看到tag为 <em>new_job</em> 的event产生后的5秒, 自动产生了一个fun值为 <em>saltutil.find_job</em>, 其arg为之前new_job的jid的event. 然后minion返回之前运行的fun值为 <em>cmd.run</em> 对应的进行运行信息(pid等信息, 已确保任务正在被执行).</p>
<p>Salt对应的处理机制是master在下发指令后,如果在设置的timeout时间内, 所有minion均返回了结果, 则直接退出. 如果达到timeout时间后, 依然有minion没有返回结果, 则自动触发一个 <em>saltutil.find_job</em> 的任务, 去所有minion上查询该任务是否在执行. 如果minion返回任务当前正在执行中, 则等待一个新的timeout周期, 如果期间所有minion均返回了结果, 则退出; 依次类推, 一直等到直到所有minion均返回结果. 如果期间在触发 <em>saltutil.find_job</em> 时minion并没有返回任务的执行状况, 且之前并没有返回结果, 则认为minion出现问题, 就会输出"Minion did not return" 字样(可以通过salt -v参数查询到).</p>
<p>从该机制中可以知道, 如果经常出现minion无法返回结果的情况, 对于某些场景如规模较大或minion高负载的情况下, 达到设置的timeout时间时, 自动触发 <em>saltutil.find_job</em> 任务, 而minion此时并没有开始运行之前下发的任务. 导致master直接认为"Minion did not return". 此时需要增大timeout的值(可以修改master的配置文件中的timeout选项)</p>
<p>同时由于master会自动触发 <em>saltutil.find_job</em> 任务, 而该任务也会记入Event系统, 所以对于如Halite等第三方系统, 执行长时间的任务时, 你会发现大量的 <em>saltutil.find_job</em> 操作, 此为正常现象, 无需处理(当然, 有洁癖的同学可能会不爽).</p>
</div>
<div class="section" id="section-3">
<h2>总结</h2>
<p>Salt提供了强大的Event系统, 第三方程序可以轻松插入Event系统, 捕获当前Salt的运行状态, 易于扩展Salt功能.</p>
</div>
ntpq peers字段说明2014-03-03T00:00:00+08:002014-03-03T00:00:00+08:00pengyaotag:pengyao.org,2014-03-03:/ntpq-peers.html<p class="first last">服务器为了保持时间的统一, 经常会部署ntpd服务. 经常使用 <tt class="docutils literal">ntpq <span class="pre">-p</span></tt> 来查看ntpd peers的运行状态信息, 但其输出中的各个字段是什么含义哪?</p>
<p>服务器为了保持时间的统一, 会部署NTPD来进行时间的同步. 对于ntpd当前的状态, 经常会通过 <tt class="docutils literal">ntpq <span class="pre">-p</span></tt> 或 通过 <tt class="docutils literal">ntpq</tt> 的console使用 <tt class="docutils literal">peers</tt> 来进行查看. 输出以下类似结果:</p>
<pre class="literal-block">
remote refid st t when poll reach delay offset jitter
==============================================================================
+42.96.167.209 202.118.1.46 2 u 32 64 377 19.047 19.838 38.791
*dns1.synet.edu. 202.118.1.46 2 u 16 64 377 120.494 48.096 25.146
+114.113.198.166 202.118.1.46 2 u 24 64 377 29.381 9.855 24.694
+61.135.250.78 223.255.185.2 2 u 24 64 377 4.793 30.086 20.347
</pre>
<p>那么每个字段代表什么含义哪?</p>
<div class="section" id="section-1">
<h2>字段含义</h2>
<div class="section" id="tally">
<h3>tally</h3>
<p>single-character code indicating current value of the select field of the peer status word</p>
<p>在每个peer最前边的一个标签字符串, 用来表示该peer的select状态.</p>
<ul class="simple">
<li><tt class="docutils literal">*</tt> : sys.peer, 表示该peer已标记为system peer, 以其变量作为系统变量.</li>
<li><tt class="docutils literal">+</tt> : candidat, 表示该peer存活并且作为候选peer.</li>
<li><tt class="docutils literal">-</tt> : outlyer, 表示该peer已经被ntp cluster算法标记为偏远的peer.</li>
<li><tt class="docutils literal">#</tt> : selected, 表示该peer存活, 但并不在前六个同步peer的范围内.</li>
<li><tt class="docutils literal">.</tt> : excess, 表示该peer不在前十个同步peer的范围内.</li>
<li><tt class="docutils literal">o</tt> : pps.peer, The peer has been declared the system peer and lends its variables to thesystem variables. However, the actual system synchronization is derived from a pulse-per-second (PPS) signal, either indirectly via the PPS reference clock driver or directly via kernel interface.</li>
<li><tt class="docutils literal">x</tt> : falsetick, The peer is discarded by the intersection algorithm as a falseticker.</li>
<li>空字符: space reject, 表示该peer标记为不可达.</li>
</ul>
</div>
<div class="section" id="remote">
<h3>remote</h3>
<p>host name (or IP number) of peer</p>
<p>该peer的主机名或IP地址, 可以在运行 <tt class="docutils literal">ntp</tt> 命令时使用 <tt class="docutils literal"><span class="pre">-n</span></tt> 选项来不进行DNS解析, 直接显示为IP地址</p>
</div>
<div class="section" id="refid">
<h3>refid</h3>
<p>association ID or kiss code</p>
<p>该peer的关联ID, 及该peer自身的 system.peer.</p>
</div>
<div class="section" id="st">
<h3>st</h3>
<p>stratum</p>
<p>该peer在NTP结构中的层级. NTP为层级结构, 数字越小表示其层级越高, 有效值是0到15.</p>
</div>
<div class="section" id="t">
<h3>t</h3>
<p>type</p>
<p>peer类型. <tt class="docutils literal">u</tt> 为单播(unicast), <tt class="docutils literal">b</tt> 为广播(broadcast), <tt class="docutils literal">l</tt> 为本地(local).</p>
</div>
<div class="section" id="when">
<h3>when</h3>
<p>sec/min/hr since last received packet</p>
<p>上次收到的该peer的ntp回应包距当前的时间.</p>
</div>
<div class="section" id="poll">
<h3>poll</h3>
<p>poll interval</p>
<p>查询间隔, 单位为秒. ntp算法会基于该peer的时间状况对该peer的查询间隔进行动态调整. 如果该peer的时间状态良好, 会在同步过程中加大该同步间隔, 反之则减小查询间隔. 在CentOS 6系统中, 默认的该查询间隔的最小值是64s, 最大值是1024s.</p>
</div>
<div class="section" id="reach">
<h3>reach</h3>
<p>reach shift register (octal)</p>
<p>标记该peer的reach计数, 该值是8进制数.</p>
<p>该值是写本文的直接原因, 因为发现之对该值的理解是错误的. 之前的理解是参照 <a class="reference external" href="http://linux.vbird.org/linux_server/0440ntp.php#server_start">鸟哥书中的描述</a> , <em>已經向上層 NTP 伺服器要求更新的次數</em>. 而实际上该值并不是计数器, 而是表示最近八次ntp查询的reach状态.</p>
<p>该值为8进制数, 由三个数组组成, 每个八进制对应3比特. 起始值是0, 之后的比特将每次进行完pool之后左移一位(如果收到ntp回应,则设置为1, 反之为0)</p>
<p>因此通常启动时, 该值的变化为: 0, 1, 3, 17, 37, 77, 177, 377</p>
<p>以本文中输出为例, 377八进制数换算成2进制为11111111, 即最近八次查询均可达.</p>
</div>
<div class="section" id="delay">
<h3>delay</h3>
<p>roundtrip delay</p>
<p>查询往返延迟, 单位是毫秒(milliseconds)</p>
</div>
<div class="section" id="offset">
<h3>offset</h3>
<p>offset</p>
<p>peer与本机时间偏差, 单位为毫秒(milliseconds)</p>
</div>
<div class="section" id="jitter">
<h3>jitter</h3>
<p>jitter</p>
<p>尚未学习该字段含义</p>
</div>
</div>
<div class="section" id="section-2">
<h2>参考链接</h2>
<ul class="simple">
<li>ntpq手册: <tt class="docutils literal">man ntpq</tt></li>
<li>ntp手册: <a class="reference external" href="http://doc.ntp.org/4.2.0/ntpq.html">http://doc.ntp.org/4.2.0/ntpq.html</a></li>
<li>NTP协议: <a class="reference external" href="http://en.wikipedia.org/wiki/Network_Time_Protocol">http://en.wikipedia.org/wiki/Network_Time_Protocol</a></li>
<li>NTP Debugging Techniques: <a class="reference external" href="http://www.fifi.org/doc/ntp-doc/html/debug.htm">http://www.fifi.org/doc/ntp-doc/html/debug.htm</a></li>
</ul>
</div>
Salt整合reclass测试2014-02-17T00:00:00+08:002014-02-17T00:00:00+08:00pengyaotag:pengyao.org,2014-02-17:/reclass-salt-01.html<p class="first last">reclass是一个外部节点分类器(External Node Classifier, ENC), 可以与自动化管理工具如PUPPET, Salt及Ansible进行整合. 就对reclass进行下学习, 看能为Salt带来什么.</p>
<div class="section" id="reclass">
<h2>reclass基本介绍</h2>
<ul class="simple">
<li>项目地址: <a class="reference external" href="https://github.com/madduck/reclass/tree">https://github.com/madduck/reclass/tree</a></li>
<li>手册地址: <a class="reference external" href="http://reclass.pantsfullofunix.net/index.html">http://reclass.pantsfullofunix.net/index.html</a></li>
</ul>
<p>reclass, 全称Recursive External Node Classification, 可以与自动化管理工具结合, 为其提供ENC服务. reclass作者认为, ENC软件应该提供如下两个功能:</p>
<ul class="simple">
<li>提供组(group)中节点(node)及组关系(group memberships)的信息</li>
<li>提供节点指定的信息, 如变量</li>
</ul>
<p>对此, reclass定义了如下四种元素:</p>
<table border="1" class="docutils">
<colgroup>
<col width="16%" />
<col width="84%" />
</colgroup>
<tbody valign="top">
<tr><td>元素</td>
<td>描述</td>
</tr>
<tr><td>node</td>
<td>一个节点, 通常是一个计算机</td>
</tr>
<tr><td>class</td>
<td>一个分类(categroy),tag,特性(feature)或角色(role), 支持嵌套和继承</td>
</tr>
<tr><td>application</td>
<td>一组行为(behaviour)</td>
</tr>
<tr><td>parameter</td>
<td>节点指定的变量,可以通过class进行继承</td>
</tr>
</tbody>
</table>
<p>reclass在继承中, 如果parent中变量不存在, 则新增,如果存在同一变量, 类型为字符串, 则会进行覆盖. 如果变量为list类型, 则进行追加</p>
</div>
<div class="section" id="reclass-1">
<h2>reclass安装</h2>
<div class="highlight"><pre><span></span>git<span class="w"> </span>clone<span class="w"> </span>https://github.com/madduck/reclass.git
<span class="nb">cd</span><span class="w"> </span>reclass
python<span class="w"> </span>setup.py<span class="w"> </span>install
</pre></div>
</div>
<div class="section" id="reclass-2">
<h2>reclass配置及测试</h2>
<p>测试目标: 通过reclass实现ntp变量的灵活扩展</p>
<p>通用信息, <em>/srv/reclass/classes/ntp-common.yml</em></p>
<div class="highlight"><pre><span></span><span class="nt">parameters</span><span class="p">:</span>
<span class="w"> </span><span class="nt">ntp</span><span class="p">:</span>
<span class="w"> </span><span class="nt">ntpserver</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0.asia.pool.ntp.org</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1.asia.pool.ntp.org</span>
</pre></div>
<p>redhat系统继承ntp-common并进行一些特殊定制, <em>/srv/reclass/classes/ntp-redhat.yml</em></p>
<div class="highlight"><pre><span></span><span class="nt">classes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ntp-common</span>
<span class="nt">parameters</span><span class="p">:</span>
<span class="w"> </span><span class="nt">ntp</span><span class="p">:</span>
<span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ntp</span>
<span class="w"> </span><span class="nt">service</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ntpd</span>
<span class="w"> </span><span class="nt">ntpserver</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2.asia.pool.ntp.org</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">3.asia.pool.ntp.org</span>
</pre></div>
<p>配置node, <em>/srv/reclass/nodes/salt-minion-01.yml</em></p>
<div class="highlight"><pre><span></span><span class="nt">classes</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ntp-redhat</span>
</pre></div>
<p>测试节点分类信息</p>
<div class="highlight"><pre><span></span>reclass<span class="w"> </span>-b<span class="w"> </span>/srv/reclass<span class="w"> </span>--nodeinfo<span class="w"> </span>salt-minion-01
</pre></div>
<p>输出结果</p>
<div class="highlight"><pre><span></span><span class="nt">__reclass__</span><span class="p">:</span>
<span class="w"> </span><span class="nt">environment</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">base</span>
<span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt-minion-01</span>
<span class="w"> </span><span class="nt">node</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt-minion-01</span>
<span class="w"> </span><span class="nt">timestamp</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">Mon Feb 17 09:29:53 2014</span>
<span class="w"> </span><span class="nt">uri</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">yaml_fs:///srv/reclass/nodes/salt-minion-01.yml</span>
<span class="nt">applications</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[]</span>
<span class="nt">classes</span><span class="p">:</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ntp-common</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ntp-redhat</span>
<span class="nt">environment</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">base</span>
<span class="nt">parameters</span><span class="p">:</span>
<span class="w"> </span><span class="nt">ntp</span><span class="p">:</span>
<span class="w"> </span><span class="nt">ntpserver</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0.asia.pool.ntp.org</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1.asia.pool.ntp.org</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2.asia.pool.ntp.org</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">3.asia.pool.ntp.org</span>
<span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ntp</span>
<span class="w"> </span><span class="nt">service</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ntpd</span>
</pre></div>
<p>从输出结果看, 与reclass手册描述一致</p>
</div>
<div class="section" id="reclasssalt">
<h2>reclass与salt整合测试</h2>
<p>Salt在0.17版本中,增加了 <a class="reference external" href="http://docs.saltstack.com/ref/tops/all/salt.tops.reclass_adapter.html">reclass的支持</a> .</p>
<p>Salt与reclass元素对应关系</p>
<table border="1" class="docutils">
<colgroup>
<col width="43%" />
<col width="57%" />
</colgroup>
<tbody valign="top">
<tr><td>reclass元素</td>
<td>Salt术语</td>
</tr>
<tr><td>nodes</td>
<td>hosts</td>
</tr>
<tr><td>classes</td>
<td>(none)</td>
</tr>
<tr><td>applications</td>
<td>states</td>
</tr>
<tr><td>parameters</td>
<td>pillar</td>
</tr>
</tbody>
</table>
<p>测试目标: 通过reclass为salt minion提供对应的ntp pillar信息</p>
<p>测试环境: Salt Master/Minion结构, 版本0.17.5</p>
<p>配置salt master, <em>/etc/salt/master</em></p>
<div class="highlight"><pre><span></span><span class="nn">...</span>
<span class="nt">reclass</span><span class="p">:</span><span class="w"> </span><span class="nl">&reclass</span>
<span class="nt">storage_type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">yaml_fs</span>
<span class="nt">inventory_base_uri</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/srv/reclass</span>
<span class="nt">master_tops</span><span class="p">:</span>
<span class="w"> </span><span class="nt">reclass</span><span class="p">:</span><span class="w"> </span><span class="nv">*reclass</span>
<span class="nt">ext_pillar</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">reclass</span><span class="p">:</span><span class="w"> </span><span class="nv">*reclass</span>
</pre></div>
<p>重启salt master</p>
<div class="highlight"><pre><span></span>service<span class="w"> </span>salt-master<span class="w"> </span>restart
</pre></div>
<p>测试salt-minion-01对应的ntp pillar</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'salt-minion-01'</span><span class="w"> </span>pillar.item<span class="w"> </span>ntp
</pre></div>
<p>输出结果</p>
<div class="highlight"><pre><span></span><span class="nt">salt-minion-01</span><span class="p">:</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">----------</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ntp</span><span class="p p-Indicator">:</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">----------</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ntpserver</span><span class="p p-Indicator">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">0.asia.pool.ntp.org</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1.asia.pool.ntp.org</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2.asia.pool.ntp.org</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">3.asia.pool.ntp.org</span>
<span class="w"> </span><span class="nt">pkg</span><span class="p">:</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ntp</span>
<span class="w"> </span><span class="nt">service</span><span class="p">:</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ntpd</span>
</pre></div>
<p>达成测试目标</p>
</div>
<div class="section" id="section-1">
<h2>总结</h2>
<p>salt pillar当前较弱, 只支持include, 并不支持extend等更高级的功能. 通过与reclass的整合, 借助reclass灵活的继承功能(支持多级继承), 为Salt提供专业的ENC服务, 弥补了pillar的不足.</p>
<p>当前reclass的功能相对较弱, 不过可以看到的是如Class subdirectories这类实用的功能已经在to-do list中, 期待reclass功能更为强大.</p>
</div>
Salt配置复杂nodegroup2014-01-16T00:00:00+08:002014-01-16T00:00:00+08:00pengyaotag:pengyao.org,2014-01-16:/salt-nodegroup-complex.html<p class="first last"><a class="reference external" href="http://saltstack.com/">SaltStack</a> 支持Nodegroup嵌套Nodegroup, 进而实现复杂Nodegroup</p>
<p>昨天拉风在群里问, 定义了A, B两个nodegroup, 是否可以定义一个nodegroup C, 包含A, B两个group, 实现nodegroup嵌套nodegroup进而实现复杂nodegroup(哈哈,各种绕口)</p>
<p>由于手册中并没有相关介绍, 就查询了下官方的issue, 找到了之前有人反馈过的 <a class="reference external" href="https://github.com/saltstack/salt/issues/2020">issue #2020</a> , tom说0.10.4已经实现了这个功能, 就在测试环境进行了测试:</p>
<p><em>/etc/salt/master.d/nodegroups.conf</em></p>
<div class="highlight"><pre><span></span><span class="nt">nodegroups</span><span class="p">:</span>
<span class="w"> </span><span class="nt">test1</span><span class="p">:</span><span class="w"> </span><span class="s">'L@salt-minion-01'</span>
<span class="w"> </span><span class="nt">test2</span><span class="p">:</span><span class="w"> </span><span class="s">'L@salt-minion-02'</span>
<span class="w"> </span><span class="nt">test</span><span class="p">:</span><span class="w"> </span><span class="s">'N@test1</span><span class="nv"> </span><span class="s">or</span><span class="nv"> </span><span class="s">N@test2’</span>
</pre></div>
<p>测试:</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span>-N<span class="w"> </span><span class="nb">test</span><span class="w"> </span>test.ping
</pre></div>
<p>输出结果:</p>
<div class="highlight"><pre><span></span><span class="nt">salt-minion-01</span><span class="p">:</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">True</span>
<span class="nt">salt-minion-02</span><span class="p">:</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">True</span>
</pre></div>
<p>从输出来看, nodegroup嵌套是支持的</p>
Tomcat Manager Text API测试2013-12-26T00:00:00+08:002013-12-26T00:00:00+08:00pengyaotag:pengyao.org,2013-12-26:/tomcat-manager-text-api-test.html<p class="first last">Tomcat Manager提供应用的状态查询,部署,维护等功能. 同时提供有Text API接口,对其进行基本测试.</p>
<div class="section" id="section-1">
<h2>基本介绍</h2>
<p>Tomcat Manager提供应用的状态查询, 部署, 维护等功能, 使用方法请参考: <a class="reference external" href="http://tomcat.apache.org/tomcat-7.0-doc/manager-howto.html">Tomcat Manager HowTo</a></p>
<p>对Tomcat Manager的Text接口进行了测试, 附上测试用例及结果分析</p>
</div>
<div class="section" id="section-2">
<h2>环境说明</h2>
<ul class="simple">
<li>Tomcat: 7.0.49</li>
<li>Manager URL: <a class="reference external" href="http://127.0.0.1:8080/manager/">http://127.0.0.1:8080/manager/</a></li>
<li>测试工具: curl</li>
</ul>
</div>
<div class="section" id="manager">
<h2>manager访问控制配置</h2>
<ul>
<li><p class="first">${CATALINA_BASE}/conf/tomcat-users.xml</p>
<div class="highlight"><pre><span></span>......
<span class="nt"><tomcat-users></span>
<span class="w"> </span><span class="nt"><user</span><span class="w"> </span><span class="na">username=</span><span class="s">"pengyao"</span><span class="w"> </span><span class="na">password=</span><span class="s">"pengyao_pass"</span><span class="w"> </span><span class="na">roles=</span><span class="s">"manager-script"</span><span class="nt">/></span>
<span class="nt"></tomcat-users></span>
</pre></div>
</li>
<li><p class="first">${CATALINA_BASE}/conf/Catalina/localhost/manager.xml</p>
<div class="highlight"><pre><span></span><span class="nt"><Context</span><span class="w"> </span><span class="na">privileged=</span><span class="s">"true"</span><span class="w"> </span><span class="na">antiResourceLocking=</span><span class="s">"false"</span><span class="w"> </span><span class="na">docBase=</span><span class="s">"${catalina.home}/webapps/manager"</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><Valve</span><span class="w"> </span><span class="na">className=</span><span class="s">"org.apache.catalina.valves.RemoteAddrValve"</span><span class="w"> </span><span class="na">allow=</span><span class="s">"127.0.0.1"</span><span class="w"> </span><span class="nt">/></span>
<span class="nt"></Context></span>
</pre></div>
</li>
<li><p class="first">配置完成后需要重启Tomcat</p>
</li>
</ul>
</div>
<div class="section" id="section-3">
<h2>测试</h2>
<div class="section" id="section-4">
<h3>查询服务信息</h3>
<ul>
<li><p class="first">Request:</p>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-u<span class="w"> </span>pengyao:pengyao_pass<span class="w"> </span>http://127.0.0.1:8080/manager/text/serverinfo
</pre></div>
</li>
<li><p class="first">Response:</p>
<pre class="literal-block">
OK - Server info
Tomcat Version: Apache Tomcat/7.0.47
OS Name: Linux
OS Version: 2.6.32-358.11.1.el6.x86_64
OS Architecture: amd64
JVM Version: 1.6.0_45-b06
JVM Vendor: Sun Microsystems Inc.
</pre>
</li>
</ul>
</div>
<div class="section" id="section-5">
<h3>获取当前已部署应用列表</h3>
<ul>
<li><p class="first">Request:</p>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-u<span class="w"> </span>pengyao:pengyao_pass<span class="w"> </span>http://127.0.0.1:8080/manager/text/list
</pre></div>
</li>
<li><p class="first">Response:</p>
<pre class="literal-block">
OK - Listed applications for virtual host localhost
/:running:0:ROOT
/manager:running:6:manager
/docs:running:0:docs
/examples:running:0:examples
/host-manager:running:0:host-manager
</pre>
</li>
</ul>
<p>结果以冒号做分隔, 第一列为context path, 第二列为当前应用的状态, 第三列为当前活跃的session数, 第四列为appBase</p>
</div>
<div class="section" id="sessionssessions">
<h3>获取应用sessions设置及当前sessions</h3>
<ul>
<li><p class="first">Request:</p>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-u<span class="w"> </span>pengyao:pengyao_pass<span class="w"> </span>http://127.0.0.1:8080/manager/text/sessions?path<span class="o">=</span>/manager
</pre></div>
</li>
<li><p class="first">Response:</p>
<pre class="literal-block">
OK - Session information for application at context path /manager
Default maximum session inactive interval 30 minutes
24 - <25 minutes: 1 sessions
26 - <27 minutes: 1 sessions
</pre>
</li>
</ul>
</div>
<div class="section" id="section-6">
<h3>启动应用</h3>
<ul>
<li><p class="first">Request:</p>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-u<span class="w"> </span>pengyao:pengyao_pass<span class="w"> </span>http://127.0.0.1:8080/manager/text/start?path<span class="o">=</span>/examples
</pre></div>
</li>
<li><p class="first">Response:</p>
<pre class="literal-block">
OK - Started application at context path /examples
</pre>
</li>
</ul>
<p>如果指定的path(context path)不存在, 则报错:</p>
<pre class="literal-block">
FAIL - No context exists for path /pengyao
</pre>
</div>
<div class="section" id="section-7">
<h3>关闭应用</h3>
<ul>
<li><p class="first">Request:</p>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-u<span class="w"> </span>pengyao:pengyao_pass<span class="w"> </span>http://127.0.0.1:8080/manager/text/stop?path<span class="o">=</span>/examples
</pre></div>
</li>
<li><p class="first">Response:</p>
<pre class="literal-block">
OK - Stopped application at context path /examples
</pre>
</li>
</ul>
</div>
<div class="section" id="section-8">
<h3>重启应用</h3>
<p>主要应用于更新了类或jar, 但没有配置 <em>reloadable</em>, 需要手动进行重启应用, 以使其生效</p>
<ul>
<li><p class="first">Request:</p>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-u<span class="w"> </span>pengyao:pengyao_pass<span class="w"> </span>http://127.0.0.1:8080/manager/text/reload?path<span class="o">=</span>/examples
</pre></div>
</li>
<li><p class="first">Response:</p>
<pre class="literal-block">
OK - Reloaded application at context path /examples
</pre>
</li>
</ul>
<p>如果应用reload前并不处于running状态, 则会报错:</p>
<pre class="literal-block">
FAIL - Encountered exception java.lang.IllegalStateException: Context with name [/examples] has not yet been started
</pre>
</div>
<div class="section" id="section-9">
<h3>卸载应用</h3>
<ul>
<li><p class="first">注意: 该操作将会删除 appBase及对应的war包, 请谨慎使用. 如果只是想暂停某应用, 请使用 <strong>关闭应用(stop)</strong> 方法</p>
</li>
<li><p class="first">Request:</p>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-u<span class="w"> </span>pengyao:pengyao_pass<span class="w"> </span>http://127.0.0.1:8080/manager/text/undeploy?path<span class="o">=</span>/examples
</pre></div>
</li>
<li><p class="first">Response:</p>
<pre class="literal-block">
OK - Undeployed application at context path /examples
</pre>
</li>
</ul>
<p>如果应用不存在,则报错:</p>
<pre class="literal-block">
FAIL - No context exists for path /pengyao
</pre>
</div>
<div class="section" id="section-10">
<h3>部署应用</h3>
<p>下载测试用例:</p>
<blockquote>
<div class="highlight"><pre><span></span>wget<span class="w"> </span>http://tomcat.apache.org/tomcat-6.0-doc/appdev/sample/sample.war<span class="w"> </span>-O<span class="w"> </span>/tmp/sample.war
</pre></div>
</blockquote>
<p><strong>以PUT方式上传war包部署应用</strong></p>
<ul>
<li><p class="first">Request:</p>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-u<span class="w"> </span>pengyao:pengyao_pass<span class="w"> </span>-T<span class="w"> </span>/tmp/sample.war<span class="w"> </span>http://127.0.0.1:8080/manager/text/deploy?path<span class="o">=</span>/sample
</pre></div>
</li>
<li><p class="first">Response:</p>
<pre class="literal-block">
OK - Deployed application at context path /sample
</pre>
</li>
</ul>
<p>此时将能在${CATALINA_BASE}/webapps/下找到sample.war 及 sample目录</p>
<p><strong>以本地(Tomcat Server本地)war包部署应用</strong></p>
<ul>
<li><p class="first">Request:</p>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-u<span class="w"> </span>pengyao:pengyao_pass<span class="w"> </span><span class="s2">"http://127.0.0.1:8080/manager/text/deploy?path=/foo&war=file:/tmp/sample.war"</span>
</pre></div>
</li>
<li><p class="first">Response:</p>
<pre class="literal-block">
OK - Deployed application at context path /foo
</pre>
</li>
</ul>
<p>此时将能在${CATALINA_BASE}/webapps/下找到foo.war 及 foo目录</p>
</div>
</div>
CentOS平台上如何加入World Community Grid参与运算2013-12-10T00:00:00+08:002013-12-10T00:00:00+08:00pengyaotag:pengyao.org,2013-12-10:/howto_join_world_community_grid_on_centos.html<p class="first last"><a class="reference external" href="http://www.worldcommunitygrid.org/">World Community Grid</a> 是一项基于互联网的公益性分布式计算项目,该项目联合分布在各地的自愿者们提供的计算资源,用于一些能够为全人类带来福音的大型科技研究项目。 当手头有CentOS操作系统的计算资源,安装上 <a class="reference external" href="http://boinc.berkeley.edu/">Boinc</a> client, 进行简单配置即可参与计算。</p>
<p><a class="reference external" href="http://www.wikipedia.org/">Wikipedia</a> 对 <a class="reference external" href="http://zh.wikipedia.org/wiki/%E4%B8%96%E7%95%8C%E5%85%AC%E5%85%B1%E7%BD%91%E6%A0%BC">World Communtiy Grid</a> 有如下描述:</p>
<pre class="literal-block">
World Community Grid,中文译名为“世界社群网格”、“世界共同体网格计划”或“世界公共网格”。是由IBM公司主持的一项基于互联网的公益性分布式计算项目,开始于2004年11月16日。该项目将联合分布于世界各地的志愿者们提供的计算资源,用于一些能为全人类带来福音的大型科学研究项目。
World Community Grid 创立之初是基于 Grid.org 的平台搭建的,之后于 2007 年开始全面迁移至开源的 BOINC 平台。World Community Grid 在底层计算平台的基础上,为具体的计算项目提供了一个更高层次的计算平台。
</pre>
<p>如果手头刚好有空闲的计算资源, 只需要安装上 <a class="reference external" href="http://boinc.berkeley.edu/">Boinc</a> client,进行简单的配置即可参与计算,贡献自己的一份力量,本文以 <a class="reference external" href="http://www.centos.org/">CentOS</a> 为例进行说明。</p>
<div class="section" id="world-community-grid">
<h2>注册World Community Grid账户</h2>
<p>访问 <a class="reference external" href="https://secure.worldcommunitygrid.org/reg/viewRegister.do">World Community Grid 注册URL</a> 进行账户注册。注册完毕后,可以在 <em>settings</em> 中设置参与哪些项目的运算,并在My Profile页面获取到 <em>Weak Account Key</em>.</p>
</div>
<div class="section" id="boinc-client">
<h2>安装配置Boinc Client</h2>
<p><a class="reference external" href="http://fedoraproject.org/wiki/EPEL/zh-cn">EPEL</a> 中已经有了 <a class="reference external" href="http://boinc.berkeley.edu/">Boinc</a> client软件包,直接进行安装。如果你还没有安装EPEL,请在进行以下操作前先进行EPEL的安装。</p>
<p><strong>安装boinc-client</strong></p>
<div class="highlight"><pre><span></span>yum<span class="w"> </span>install<span class="w"> </span>boinc-client
</pre></div>
<p><strong>配置boinc-client</strong></p>
<div class="highlight"><pre><span></span><span class="nb">echo</span><span class="w"> </span><span class="s1">'BOINCOPTS="--daemon --attach_project www.worldcommunitygrid.org/ weak_account_key"'</span><span class="w"> </span>>>/etc/sysconfig/boinc-client
</pre></div>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">请将命令中的 <em>weak_account_key</em> 替换为之前在My Profile页面获取到的 <em>Weak Account Key</em></p>
</div>
<p><strong>启动boinc-client</strong></p>
<div class="highlight"><pre><span></span>chkconfig<span class="w"> </span>boinc-client<span class="w"> </span>on
service<span class="w"> </span>boinc-client<span class="w"> </span>start
</pre></div>
</div>
<div class="section" id="section-1">
<h2>查看已参与运算的设备信息</h2>
<p>安装配置boinc-client并运行后,访问 <a class="reference external" href="http://www.worldcommunitygrid.org/">World Community Grid</a> 的 <a class="reference external" href="https://secure.worldcommunitygrid.org/ms/device/viewDevices.do">Device Manager</a> 即可查询到当前已参与运算的设备信息。</p>
<p>Technology solving problems!</p>
</div>
Salt-API安装配置及使用2013-11-27T00:00:00+08:002013-11-27T00:00:00+08:00pengyaotag:pengyao.org,2013-11-27:/salt-api-deploy-and-use.html<p class="first last"><a class="reference external" href="http://saltstack.com/">SaltStack</a> 官方提供有REST API格式的 <a class="reference external" href="https://github.com/saltstack/salt-api">salt-api</a> 项目, 与第三方系统集成将变得非常简单。 本文将讲述如何安装配置及使用Salt-API。</p>
<p><a class="reference external" href="http://saltstack.com/">SaltStack</a> 官方提供有REST API格式的 <a class="reference external" href="https://github.com/saltstack/salt-api">salt-api</a> 项目,将使Salt与第三方系统集成变得尤为简单。本文讲带你了解如何安装配置Salt-API, 如何利用Salt-API获取想要的信息。</p>
<div class="section" id="section-1">
<h2>前置阅读</h2>
<ul class="simple">
<li><a class="reference external" href="http://salt-api.readthedocs.org/en/latest/">salt-api手册</a></li>
<li><a class="reference external" href="http://docs.saltstack.com/topics/eauth/index.html">Salt External Authentication System</a></li>
</ul>
</div>
<div class="section" id="section-2">
<h2>环境说明</h2>
<ul class="simple">
<li>操作系统环境: CentOS 6.4,已配置EPEL源</li>
<li>Salt Master/Minion版本: 0.17.2, Master IP地址为 <em>192.168.3</em>, 用于本次测试的Minion ID为 <em>minion-01.example.com</em></li>
</ul>
</div>
<div class="section" id="section-3">
<h2>开工</h2>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">以下操作如非特别注明,均在Master上进行</p>
</div>
<div class="section" id="salt-api-1">
<h3>安装Salt-API</h3>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">当前EPEL中的salt-api版本为0.8.2, 存在几处bug, 本文讲使用pip方式安装0.8.3版本</p>
</div>
<div class="highlight"><pre><span></span><span class="c1"># 安装salt-api</span>
pip<span class="w"> </span>install<span class="w"> </span>salt-api
<span class="c1"># 下载服务维护脚本</span>
wget<span class="w"> </span>https://raw.github.com/saltstack/salt-api/develop/pkg/rpm/salt-api<span class="w"> </span>-O<span class="w"> </span>/etc/init.d/salt-api
chmod<span class="w"> </span>+x<span class="w"> </span>/etc/init.d/salt-api
chkconfig<span class="w"> </span>salt-api<span class="w"> </span>on
</pre></div>
</div>
<div class="section" id="salt-api-2">
<h3>配置Salt-API</h3>
<div class="section" id="ssl">
<h4>生成自签名证书(用于ssl)</h4>
<div class="highlight"><pre><span></span><span class="nb">cd</span><span class="w"> </span>/etc/pki/tls/certs
<span class="c1"># 生成自签名证书, 过程中需要输入key密码及RDNs</span>
make<span class="w"> </span>testcert
<span class="nb">cd</span><span class="w"> </span>/etc/pki/tls/private/
<span class="c1"># 解密key文件,生成无密码的key文件, 过程中需要输入key密码,该密码为之前生成证书时设置的密码</span>
openssl<span class="w"> </span>rsa<span class="w"> </span>-in<span class="w"> </span>localhost.key<span class="w"> </span>-out<span class="w"> </span>localhost_nopass.key
</pre></div>
</div>
<div class="section" id="salt-api-3">
<h4>Salt-API配置</h4>
<ul class="simple">
<li>创建用于salt-api的用户</li>
</ul>
<div class="highlight"><pre><span></span>useradd<span class="w"> </span>-M<span class="w"> </span>-s<span class="w"> </span>/sbin/nologin<span class="w"> </span>pengyao
<span class="nb">echo</span><span class="w"> </span><span class="s2">"pengyao_pass"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>passwd<span class="w"> </span>pengyao<span class="w"> </span>—stdin
</pre></div>
<ul class="simple">
<li>配置eauth, <em>/etc/salt/master.d/eauth.conf</em></li>
</ul>
<div class="highlight"><pre><span></span><span class="nt">external_auth</span><span class="p">:</span>
<span class="w"> </span><span class="nt">pam</span><span class="p">:</span>
<span class="w"> </span><span class="nt">pengyao</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">.*</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">'@wheel'</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="s">'@runner'</span>
</pre></div>
<ul class="simple">
<li>配置Salt-API, <em>/etc/salt/master.d/api.conf</em></li>
</ul>
<div class="highlight"><pre><span></span><span class="nt">rest_cherrypy</span><span class="p">:</span>
<span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">8000</span>
<span class="w"> </span><span class="nt">ssl_crt</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/etc/pki/tls/certs/localhost.crt</span>
<span class="w"> </span><span class="nt">ssl_key</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/etc/pki/tls/private/localhost_nopass.key</span>
</pre></div>
<ul class="simple">
<li>启动Salt-API</li>
</ul>
<div class="highlight"><pre><span></span>service<span class="w"> </span>salt-api<span class="w"> </span>start
</pre></div>
</div>
</div>
</div>
<div class="section" id="salt-api-4">
<h2>Salt-API使用</h2>
<ul class="simple">
<li>测试工具为操作系统自带的 <em>curl</em></li>
</ul>
<div class="section" id="login">
<h3>Login</h3>
<ul class="simple">
<li>Request</li>
</ul>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-k<span class="w"> </span>https://192.168.38.10:8000/login<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Accept: application/x-yaml"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="nv">username</span><span class="o">=</span><span class="s1">'pengyao'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="nv">password</span><span class="o">=</span><span class="s1">'pengyao_pass'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="nv">eauth</span><span class="o">=</span><span class="s1">'pam'</span>
</pre></div>
<ul class="simple">
<li>Response</li>
</ul>
<div class="highlight"><pre><span></span><span class="nt">return</span><span class="p">:</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">eauth</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pam</span>
<span class="w"> </span><span class="nt">expire</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1385579710.806725</span>
<span class="w"> </span><span class="nt">perms</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">.*</span>
<span class="w"> </span><span class="nt">start</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">1385536510.8067241</span>
<span class="w"> </span><span class="nt">token</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">784ee23c63794576a50ca5d3d890eb71efb0de6f</span>
<span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pengyao</span>
</pre></div>
<p>其中 <em>token</em> 后边的串为认证成功后获取的token串,之后可以不用再次输入密码,直接使用本Token即可</p>
</div>
<div class="section" id="minion-minion-01-example-com">
<h3>查询Minion(minion-01.example.com)的信息</h3>
<ul class="simple">
<li>Request</li>
</ul>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-k<span class="w"> </span>https://192.168.38.10:8000/minions/minion-01.example.com<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Accept: application/x-yaml"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: 8e211da5d6bbb51fbffe6468a3ca0c6a24b3e535"</span>
</pre></div>
<p>其中 <em>X-Auth-Token</em> 后边的串为之前Login获取到的Token串, 如果请求的URL不包含 <em>minion-01.example.com</em> ,则请求的为所有Minion的信息</p>
<ul class="simple">
<li>Response</li>
</ul>
<div class="highlight"><pre><span></span><span class="nt">return</span><span class="p">:</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">minion-01.example.com</span><span class="p">:</span>
<span class="w"> </span><span class="nt">cpu_flags</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">fpu</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">vme</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">de</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">......</span>
</pre></div>
</div>
<div class="section" id="job">
<h3>job管理</h3>
<div class="section" id="jobs">
<h4>获取缓存的jobs列表</h4>
<ul class="simple">
<li>Request</li>
</ul>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-k<span class="w"> </span>https://192.168.38.10:8000/jobs/<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Accept: application/x-yaml"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: 8e211da5d6bbb51fbffe6468a3ca0c6a24b3e535"</span>
</pre></div>
<ul class="simple">
<li>Response</li>
</ul>
<div class="highlight"><pre><span></span><span class="nt">return</span><span class="p">:</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="s">'20131127065003726179'</span><span class="p p-Indicator">:</span>
<span class="w"> </span><span class="nt">Arguments</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[]</span>
<span class="w"> </span><span class="nt">Function</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">test.ping</span>
<span class="w"> </span><span class="nt">Start Time</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2013, Nov 27 06:50:03.726179</span>
<span class="w"> </span><span class="nt">Target</span><span class="p">:</span><span class="w"> </span><span class="s">'*'</span>
<span class="w"> </span><span class="nt">Target-type</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">glob</span>
<span class="w"> </span><span class="nt">User</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">sudo_vagrant</span>
</pre></div>
</div>
<div class="section" id="job-1">
<h4>查询指定的job</h4>
<ul class="simple">
<li>Request</li>
</ul>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-k<span class="w"> </span>https://192.168.38.10:8000/jobs/20131127065003726179<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Accept: application/x-yaml"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: 8e211da5d6bbb51fbffe6468a3ca0c6a24b3e535"</span>
</pre></div>
<ul class="simple">
<li>Response</li>
</ul>
<div class="highlight"><pre><span></span><span class="nt">return</span><span class="p">:</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">minion-01.example.com</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
</pre></div>
</div>
</div>
<div class="section" id="section-4">
<h3>远程执行模块</h3>
<ul class="simple">
<li>Request</li>
</ul>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-k<span class="w"> </span>https://192.168.38.10:8000/<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Accept: application/x-yaml"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: 8e211da5d6bbb51fbffe6468a3ca0c6a24b3e535"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="nv">client</span><span class="o">=</span><span class="s1">'local'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="nv">tgt</span><span class="o">=</span><span class="s1">'*'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="nv">fun</span><span class="o">=</span><span class="s1">'test.ping'</span>
</pre></div>
<p>也可以请求 <em>https://192.168.38.10:8000/run</em> ,不过该方法为一次性使用,无法使用Token, 只能使用username和password</p>
<ul class="simple">
<li>Response:</li>
</ul>
<div class="highlight"><pre><span></span><span class="nt">return</span><span class="p">:</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">minion-01.example.com</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
</pre></div>
</div>
<div class="section" id="runner">
<h3>运行runner</h3>
<ul class="simple">
<li>Request</li>
</ul>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-k<span class="w"> </span>https://192.168.38.10:8000/<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Accept: application/x-yaml"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: 8e211da5d6bbb51fbffe6468a3ca0c6a24b3e535"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="nv">client</span><span class="o">=</span><span class="s1">'runner'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="nv">fun</span><span class="o">=</span><span class="s1">'manage.status'</span>
</pre></div>
<ul class="simple">
<li>Response</li>
</ul>
<div class="highlight"><pre><span></span><span class="nt">return</span><span class="p">:</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">down</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[]</span>
<span class="w"> </span><span class="nt">up</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">minion-01.example.com</span>
</pre></div>
</div>
<div class="section" id="wheel">
<h3>运行wheel</h3>
<ul class="simple">
<li>注意: 由于当前版本的Salt Master有一处bug, 导致wheel的结果无法返回(<a class="reference external" href="https://groups.google.com/forum/#!topic/salt-users/9HcZ6R7MB0g">https://groups.google.com/forum/#!topic/salt-users/9HcZ6R7MB0g</a>),官方在最新的代码中已经修复,使用时需要使用github中最新的salt代码</li>
<li>Request(例子为查询所有的minion key列表)</li>
</ul>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-k<span class="w"> </span>https://localhost:8000/<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Accept: application/x-yaml"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: 8e211da5d6bbb51fbffe6468a3ca0c6a24b3e535"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="nv">client</span><span class="o">=</span><span class="s1">'wheel'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="nv">fun</span><span class="o">=</span><span class="s1">'key.list_all'</span>
</pre></div>
<ul class="simple">
<li>Response</li>
</ul>
<div class="highlight"><pre><span></span><span class="nt">return</span><span class="p">:</span>
<span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">data</span><span class="p">:</span>
<span class="w"> </span><span class="nt">_stamp</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">2013-12-23_04:54:22.483159</span>
<span class="w"> </span><span class="nt">fun</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">wheel.key.list_all</span>
<span class="w"> </span><span class="nt">jid</span><span class="p">:</span><span class="w"> </span><span class="s">'20131223045422481844'</span>
<span class="w"> </span><span class="nt">return</span><span class="p">:</span>
<span class="w"> </span><span class="nt">local</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">master.pem</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">master.pub</span>
<span class="w"> </span><span class="nt">minions</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">minion-01.example.com</span>
<span class="w"> </span><span class="nt">minions_pre</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[]</span>
<span class="w"> </span><span class="nt">minions_rejected</span><span class="p">:</span><span class="w"> </span><span class="p p-Indicator">[]</span>
<span class="w"> </span><span class="w w-Error"> </span><span class="nt">success</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">true</span>
<span class="w"> </span><span class="nt">tag</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt/wheel/20131223045422481844</span>
<span class="w"> </span><span class="nt">user</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pengyao</span>
<span class="w"> </span><span class="w w-Error"> </span><span class="nt">tag</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt/wheel/20131223045422481844</span>
</pre></div>
</div>
<div class="section" id="targeting">
<h3>Targeting</h3>
<p>谢谢 <em>苦咖啡</em> 提供</p>
<p>如果想在api中使用salt的 <a class="reference external" href="http://docs.saltstack.com/topics/targeting/">Targeting</a> 功能,可以在Request的Post Data中增加 <em>expr_form</em> (默认是 <em>glob</em> )及值即可:</p>
<p>依然以curl为例:</p>
<div class="highlight"><pre><span></span>curl<span class="w"> </span>-k<span class="w"> </span>https://192.168.38.10:8000/<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Accept: application/x-yaml"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: 8e211da5d6bbb51fbffe6468a3ca0c6a24b3e535"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="nv">client</span><span class="o">=</span><span class="s1">'local'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="nv">tgt</span><span class="o">=</span><span class="s1">'webcluster'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="nv">expr_form</span><span class="o">=</span><span class="s1">'nodegroup'</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="nv">fun</span><span class="o">=</span><span class="s1">'test.ping'</span>
</pre></div>
<p>将利用 <a class="reference external" href="http://docs.saltstack.com/topics/targeting/nodegroups.html">nodegroup</a> 匹配到名为 <em>webcluster</em> 的target。</p>
</div>
</div>
<div class="section" id="section-5">
<h2>总结</h2>
<p>Salt API几乎涵盖了所有的salt操作,功能强劲,尤其是需要salt和第三方系统集成的场景,值得拥有。</p>
</div>
基于SaltStack完成LVS的配置管理2013-11-24T00:00:00+08:002013-11-24T00:00:00+08:00pengyaotag:pengyao.org,2013-11-24:/howto_configure_linux_virtual_server_using_saltstack.html<p class="first last">SaltStack最新代码中已经包含了对LVS(Linux Virutal Server)的支持, 本文将简要描述如何基于SaltStack完成LVS Loadblance(DR)及RealServer的配置管理</p>
<p>之前由于工作需求,编写了SaltStack的 <a class="reference external" href="https://github.com/pengyao/salt/blob/develop/salt/modules/lvs.py">LVS远程执行模块</a> , <a class="reference external" href="https://github.com/pengyao/salt/blob/develop/salt/states/lvs_service.py">LVS service状态管理模块</a> 及 <a class="reference external" href="https://github.com/pengyao/salt/blob/develop/salt/states/lvs_server.py">LVS server状态管理模块</a> ,并 <a class="reference external" href="https://github.com/saltstack/salt/pull/8741">提交给了SaltStack官方</a>,现已合并至官方代码中,本文将描述如何基于SaltStack完成LVS Loadblance(DR)及RealServer的配置管理.</p>
<div class="section" id="section-1">
<h2>前置阅读</h2>
<ul class="simple">
<li><a class="reference external" href="http://blog.csdn.net/justlinux2010/article/details/8539205">LVS-DR模式配置详解</a> ,需要注意的是,LVS-DR方式工作在数据链路层,文中描述需要开启ip_forward,其实没有必要, 详情见 <a class="reference external" href="http://zh.linuxvirtualserver.org/node/2585">LVS DR模式原理剖析</a></li>
</ul>
</div>
<div class="section" id="section-2">
<h2>环境说明</h2>
<ul class="simple">
<li>三台服务器用于LVS集群,其中主机名为lvs的担当的角色为loadblance,对应的IP地址为192.168.36.10;主机名为web-01和web-02的主机担当的角色为RealServer, 对应的IP地址分别为192.168.36.11及192.168.36.12</li>
<li>LVS VIP: 192.168.36.33, Port: 80, VIP绑定在lvs的eth1口</li>
<li>最最重要的是loadblance主机为Linux,并已安装ipvsadm, Windows/Unix等主机的同学请绕过吧,这不是我的错......</li>
</ul>
</div>
<div class="section" id="section-3">
<h2>开工</h2>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">以下所有操作均在Master上进行</p>
</div>
<div class="section" id="saltstack-lvs">
<h3>配置SaltStack LVS模块</h3>
<ul class="simple">
<li>如果使用的Salt版本已经包含了lvs模块,请忽略本节内容,测试方法:</li>
</ul>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'lvs'</span><span class="w"> </span>cmd.run<span class="w"> </span><span class="s2">"python -c 'import salt.modules.lvs'"</span>
</pre></div>
<p>如果输出有 <em>ImportError</em> 字样,则表示模块没有安装,需要进行如下操作:</p>
<div class="highlight"><pre><span></span><span class="nb">test</span><span class="w"> </span>-d<span class="w"> </span>/srv/salt/_modules<span class="w"> </span><span class="o">||</span><span class="w"> </span>mkdir<span class="w"> </span>/srv/salt/_modules
<span class="nb">test</span><span class="w"> </span>-d<span class="w"> </span>/srv/salt/_states<span class="w"> </span><span class="o">||</span><span class="w"> </span>mkdir<span class="w"> </span>/srv/salt/_states
wget<span class="w"> </span>https://raw.github.com/saltstack/salt/develop/salt/modules/lvs.py<span class="w"> </span>-O<span class="w"> </span>/srv/salt/_modules/lvs.py
wget<span class="w"> </span>https://raw.github.com/saltstack/salt/develop/salt/states/lvs_service.py<span class="w"> </span>-O<span class="w"> </span>/srv/salt/_states/lvs_service.py
wget<span class="w"> </span>https://raw.github.com/saltstack/salt/develop/salt/states/lvs_server.py<span class="w"> </span>-O<span class="w"> </span>/srv/salt/_states/lvs_server.py
</pre></div>
</div>
<div class="section" id="pillar">
<h3>配置pillar</h3>
<p><em>/srv/pillar/lvs/loadblance.sls</em></p>
<div class="highlight"><pre><span></span><span class="nt">lvs-loadblance</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">lvstest</span>
<span class="w"> </span><span class="nt">vip</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">192.168.36.33</span>
<span class="w"> </span><span class="nt">vip-nic</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">eth1</span>
<span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">80</span>
<span class="w"> </span><span class="nt">protocol</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">tcp</span>
<span class="w"> </span><span class="nt">scheduler</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">wlc</span>
<span class="w"> </span><span class="nt">realservers</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">web-01</span>
<span class="w"> </span><span class="nt">ip</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">192.168.36.11</span>
<span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">80</span>
<span class="w"> </span><span class="nt">packet_forward_method</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">dr</span>
<span class="w"> </span><span class="nt">weight</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">10</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">web-02</span>
<span class="w"> </span><span class="nt">ip</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">192.168.36.12</span>
<span class="w"> </span><span class="nt">port</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">80</span>
<span class="w"> </span><span class="nt">packet_forward_method</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">dr</span>
<span class="w"> </span><span class="nt">weight</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">30</span>
</pre></div>
<p><em>/srv/pillar/lvs/realserver.sls</em></p>
<div class="highlight"><pre><span></span><span class="nt">lvs-realserver</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">name</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">lvstest</span>
<span class="w"> </span><span class="nt">vip</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">192.168.36.33</span>
</pre></div>
<p><em>/srv/pillar/top.sls</em></p>
<div class="highlight"><pre><span></span><span class="nt">base</span><span class="p">:</span>
<span class="w"> </span><span class="s">'lvs'</span><span class="p p-Indicator">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">lvs.loadblance</span>
<span class="w"> </span><span class="s">'web-0*'</span><span class="p p-Indicator">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">lvs.realserver</span>
</pre></div>
</div>
<div class="section" id="states">
<h3>编写States</h3>
<p><em>/srv/salt/lvs/loadblance.sls</em></p>
<div class="highlight"><pre><span></span><span class="x"># config lvs</span>
<span class="cp">{%</span> <span class="k">if</span> <span class="s1">'lvs-loadblance'</span> <span class="k">in</span> <span class="nv">pillar</span> <span class="cp">%}</span>
<span class="cp">{%</span> <span class="k">for</span> <span class="nv">each_lvs</span> <span class="k">in</span> <span class="nv">pillar</span><span class="o">[</span><span class="s1">'lvs-loadblance'</span><span class="o">]</span> <span class="cp">%}</span>
<span class="x"># config lvs vip</span>
<span class="cp">{{</span><span class="nv">each_lvs</span><span class="o">[</span><span class="s1">'name'</span><span class="o">]</span><span class="cp">}}</span><span class="x">-vip:</span>
<span class="x"> network.managed:</span>
<span class="x"> - name: </span><span class="cp">{{</span><span class="nv">each_lvs</span><span class="o">[</span><span class="s1">'vip-nic'</span><span class="o">]</span> <span class="o">+</span> <span class="s2">":"</span> <span class="o">+</span> <span class="nv">each_lvs</span><span class="o">[</span><span class="s1">'name'</span><span class="o">]</span><span class="cp">}}</span>
<span class="x"> - enable: True</span>
<span class="x"> - type: eth</span>
<span class="x"> - proto: none</span>
<span class="x"> - ipaddr: </span><span class="cp">{{</span><span class="nv">each_lvs</span><span class="o">[</span><span class="s1">'vip'</span><span class="o">]</span><span class="cp">}}</span>
<span class="x"> - netmask: 255.255.255.255</span>
<span class="cp">{%</span> <span class="k">set</span> <span class="nv">service_address</span> <span class="o">=</span> <span class="nv">each_lvs</span><span class="o">[</span><span class="s1">'vip'</span><span class="o">]</span> <span class="o">+</span> <span class="s2">":"</span> <span class="o">+</span> <span class="nv">each_lvs</span><span class="o">[</span><span class="s1">'port'</span><span class="o">]|</span><span class="nf">string</span><span class="o">()</span> <span class="cp">%}</span>
<span class="cp">{{</span><span class="nv">each_lvs</span><span class="o">[</span><span class="s1">'name'</span><span class="o">]</span><span class="cp">}}</span><span class="x">-service:</span>
<span class="x"> lvs_service.present:</span>
<span class="x"> - protocol: </span><span class="cp">{{</span><span class="nv">each_lvs</span><span class="o">[</span><span class="s1">'protocol'</span><span class="o">]</span><span class="cp">}}</span>
<span class="x"> - service_address: </span><span class="cp">{{</span><span class="nv">service_address</span><span class="cp">}}</span>
<span class="x"> - scheduler: </span><span class="cp">{{</span><span class="nv">each_lvs</span><span class="o">[</span><span class="s1">'scheduler'</span><span class="o">]</span><span class="cp">}}</span>
<span class="cp">{%</span> <span class="k">for</span> <span class="nv">each_rs</span> <span class="k">in</span> <span class="nv">each_lvs</span><span class="o">[</span><span class="s1">'realservers'</span><span class="o">]</span> <span class="cp">%}</span>
<span class="cp">{%</span> <span class="k">set</span> <span class="nv">server_address</span> <span class="o">=</span> <span class="nv">each_rs</span><span class="o">[</span><span class="s1">'ip'</span><span class="o">]</span> <span class="o">+</span> <span class="s2">":"</span> <span class="o">+</span> <span class="nv">each_rs</span><span class="o">[</span><span class="s1">'port'</span><span class="o">]|</span><span class="nf">string</span><span class="o">()</span> <span class="cp">%}</span>
<span class="cp">{{</span><span class="nv">each_rs</span><span class="o">[</span><span class="s1">'name'</span><span class="o">]</span><span class="cp">}}</span><span class="x">-server:</span>
<span class="x"> lvs_server.present:</span>
<span class="x"> - protocol: </span><span class="cp">{{</span><span class="nv">each_lvs</span><span class="o">[</span><span class="s1">'protocol'</span><span class="o">]</span><span class="cp">}}</span>
<span class="x"> - service_address: </span><span class="cp">{{</span><span class="nv">service_address</span><span class="cp">}}</span>
<span class="x"> - server_address: </span><span class="cp">{{</span><span class="nv">server_address</span><span class="cp">}}</span>
<span class="x"> - packet_forward_method: </span><span class="cp">{{</span><span class="nv">each_rs</span><span class="o">[</span><span class="s1">'packet_forward_method'</span><span class="o">]</span><span class="cp">}}</span>
<span class="x"> - weight: </span><span class="cp">{{</span><span class="nv">each_rs</span><span class="o">[</span><span class="s1">'weight'</span><span class="o">]</span><span class="cp">}}</span>
<span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span>
<span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span>
<span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span>
</pre></div>
<p><em>/srv/salt/lvs/realserver.sls</em></p>
<div class="highlight"><pre><span></span><span class="x"># ignore arp</span>
<span class="x">net.ipv4.conf.all.arp_ignore:</span>
<span class="x"> sysctl.present:</span>
<span class="x"> - value: 1</span>
<span class="x">net.ipv4.conf.lo.arp_ignore:</span>
<span class="x"> sysctl.present:</span>
<span class="x"> - value: 1</span>
<span class="x">net.ipv4.conf.all.arp_announce:</span>
<span class="x"> sysctl.present:</span>
<span class="x"> - value: 2</span>
<span class="x">net.ipv4.conf.lo.arp_announce:</span>
<span class="x"> sysctl.present:</span>
<span class="x"> - value: 2</span>
<span class="x"># config lvs vip</span>
<span class="cp">{%</span> <span class="k">if</span> <span class="s1">'lvs-realserver'</span> <span class="k">in</span> <span class="nv">pillar</span> <span class="cp">%}</span>
<span class="cp">{%</span> <span class="k">for</span> <span class="nv">each_lvs</span> <span class="k">in</span> <span class="nv">pillar</span><span class="o">[</span><span class="s1">'lvs-realserver'</span><span class="o">]</span> <span class="cp">%}</span>
<span class="x">lvs-vip:</span>
<span class="x"> network.managed:</span>
<span class="x"> - name: </span><span class="cp">{{</span><span class="s2">"lo"</span> <span class="o">+</span> <span class="s2">":"</span> <span class="o">+</span> <span class="nv">each_lvs</span><span class="o">[</span><span class="s1">'name'</span><span class="o">]</span><span class="cp">}}</span>
<span class="x"> - enable: True</span>
<span class="x"> - type: eth</span>
<span class="x"> - proto: none</span>
<span class="x"> - ipaddr: </span><span class="cp">{{</span><span class="nv">each_lvs</span><span class="o">[</span><span class="s1">'vip'</span><span class="o">]</span><span class="cp">}}</span>
<span class="x"> - netmask: 255.255.255.255</span>
<span class="cp">{%</span> <span class="k">endfor</span> <span class="cp">%}</span>
<span class="cp">{%</span> <span class="k">endif</span> <span class="cp">%}</span>
</pre></div>
<ul class="simple">
<li>/srv/salt/top.sls*</li>
</ul>
<div class="highlight"><pre><span></span><span class="nt">base</span><span class="p">:</span>
<span class="w"> </span><span class="s">'lvs'</span><span class="p p-Indicator">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">lvs.loadblance</span>
<span class="w"> </span><span class="s">'web-0*'</span><span class="p p-Indicator">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">lvs.realserver</span>
</pre></div>
</div>
<div class="section" id="section-4">
<h3>应用配置</h3>
<p>如果之前进行 <em>配置LVS模块</em> 的操作,需要进行同步模块的操作:</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'lvs*'</span><span class="w"> </span>saltutil.sync_all
</pre></div>
<p>应用LVS配置:</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>state.highstate
</pre></div>
<p>查看LVS当前状态:</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'lvs'</span><span class="w"> </span>lvs.list
</pre></div>
</div>
<div class="section" id="section-5">
<h3>总结</h3>
<p>通过SaltStack LVS模块,可以快速的查询LVS状态,执行LVS常用指令及完成LVS的配置管理。如有需要调整RealServer规则或添加新的RealServer, 只需要修改 <em>/srv/pillar/lvs/loadblance.sls</em> ,然后应用配置即可.</p>
<p>本文中所用到的代码已经上传至github,传送门: <a class="reference external" href="https://github.com/pengyao/salt-lvs">https://github.com/pengyao/salt-lvs</a></p>
</div>
</div>
基于Salt Master/Minions快速构建Salt SSH环境2013-11-08T00:00:00+08:002013-11-08T00:00:00+08:00pengyaotag:pengyao.org,2013-11-08:/howto_setup_salt_ssh_from_minion.html<p class="first last">Salt 0.17版本重要的特性是引入了Salt SSH系统,本文基于已有的SaltStack Master/Minions环境,快速构建Salt SSH维护环境, 提供Salt多重维护方式.</p>
<p>Salt 0.17版本已发布,该版本中重要的特性是引入了Salt SSH系统,提供了无需Minion、基于SSH的维护方式。原有的Salt维护环境已经初具规模,再手动重新构建Salt SSH环境成本较高。偷懒是人的天性,利用原有SaltStack Master/Minions环境,如何快速构建新的Salt SSH维护环境将是本文的主题.</p>
<div class="section" id="section-1">
<h2>前置阅读</h2>
<p>预则立,不预则废。阅读本文前先阅读如下文章:</p>
<ul class="simple">
<li>Salt 0.17 Release Note: <a class="reference external" href="http://docs.saltstack.com/topics/releases/0.17.0.html">http://docs.saltstack.com/topics/releases/0.17.0.html</a></li>
<li>Salt SSH: <a class="reference external" href="http://docs.saltstack.com/topics/ssh/">http://docs.saltstack.com/topics/ssh/</a></li>
<li>Salt Rosters: <a class="reference external" href="http://docs.saltstack.com/topics/ssh/roster.html">http://docs.saltstack.com/topics/ssh/roster.html</a></li>
</ul>
</div>
<div class="section" id="section-2">
<h2>环境说明</h2>
<ul class="simple">
<li>Minion版本: 本文会采用 <a class="reference external" href="http://docs.saltstack.com/topics/mine/">Salt Mine</a> 获取已有的Minion ID及IP地址,由于Salt Mine为0.15.0引入的新功能,所以需要保证Minion的版本等于或高于0.15.0</li>
<li>Master的安装采用EPEL仓库yum方式</li>
<li>所有minion端sshd服务已启动,并允许Master访问</li>
<li>Master所在服务器上同时安装有Minion并运行Master进行管理, 对应的Minion ID为 <em>salt</em></li>
<li>Salt file_roots目录为 <em>/srv/salt/</em> , pillar_roots目录为 <em>/srv/pillar/</em></li>
</ul>
</div>
<div class="section" id="section-3">
<h2>开工</h2>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">以下所有操作在Master端进行</p>
</div>
<div class="section" id="salt-sshkey">
<h3>创建用于Salt SSH环境的用户及key认证管理环境</h3>
<p>生成Master SSH key</p>
<div class="highlight"><pre><span></span><span class="c1">## 创建master ssh key目录</span>
mkdir<span class="w"> </span>/etc/salt/pki/master/ssh/
<span class="c1">## 生成Master SSH key</span>
<span class="nb">cd</span><span class="w"> </span>/etc/salt/pki/master/ssh/
ssh-keygen<span class="w"> </span>-t<span class="w"> </span>rsa<span class="w"> </span>-P<span class="w"> </span><span class="s2">""</span><span class="w"> </span>-f<span class="w"> </span>salt-ssh.rsa
<span class="c1">## 复制master public key至 salt fileserver</span>
cp<span class="w"> </span>/etc/salt/pki/master/ssh/salt-ssh.rsa.pub<span class="w"> </span>/srv/salt/salt/files/salt-ssh.rsa.pub
</pre></div>
<p>编写用于Salt SSH管理的用户及key认证状态管理文件, <em>/srv/salt/salt/ssh/init.sls</em></p>
<div class="highlight"><pre><span></span><span class="nt">salt-user</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">{</span><span class="c1"># salt user #}</span>
<span class="nt"> user.present</span><span class="p">:</span>
<span class="nt"> - name</span><span class="p">:</span><span class="w"> </span><span class="nv">salt</span>
<span class="w"> </span><span class="p p-Indicator">{</span><span class="c1"># salt user sudoer #}</span>
<span class="nt"> file.managed</span><span class="p">:</span>
<span class="nt"> - name</span><span class="p">:</span><span class="w"> </span><span class="nv">/etc/sudoers.d/salt</span>
<span class="w"> </span><span class="nv">- source</span><span class="p p-Indicator">:</span><span class="w"> </span><span class="nv">salt</span><span class="p p-Indicator">:</span><span class="nv">//salt/files/etc/sudoers.d/salt</span>
<span class="w"> </span><span class="nv">- require</span><span class="p p-Indicator">:</span>
<span class="nt"> - user</span><span class="p">:</span><span class="w"> </span><span class="nv">salt-user</span>
<span class="nv">salt-master-key</span><span class="p p-Indicator">:</span>
<span class="nt"> ssh_auth.present</span><span class="p">:</span>
<span class="nt"> - user</span><span class="p">:</span><span class="w"> </span><span class="nv">salt</span>
<span class="w"> </span><span class="nv">- source</span><span class="p p-Indicator">:</span><span class="w"> </span><span class="nv">salt</span><span class="p p-Indicator">:</span><span class="nv">//salt/files/salt-ssh.rsa.pub</span>
<span class="w"> </span><span class="nv">- require</span><span class="p p-Indicator">:</span>
<span class="nt"> - user</span><span class="p">:</span><span class="w"> </span><span class="nv">salt-user</span>
</pre></div>
<p><em>salt</em> 用户对应的sudoer文件 <em>/srv/salt/salt/files/etc/sudoers.d/salt</em>:</p>
<pre class="literal-block">
Defaults:salt !requiretty
salt ALL=(ALL) NOPASSWD: ALL
</pre>
<p>应用状态</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>state.sls<span class="w"> </span>salt.ssh
</pre></div>
</div>
<div class="section" id="mine-minion-idip">
<h3>配置Mine,以获取Minion id及IP地址</h3>
<p>配置Salt Mine, <em>/srv/pillar/salt/mine.sls</em></p>
<div class="highlight"><pre><span></span><span class="nt">mine_functions</span><span class="p">:</span>
<span class="w"> </span><span class="nt">network.ip_addrs</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">eth0</span>
</pre></div>
<p>配置pillar top.sls, <em>/srv/pillar/top.sls</em></p>
<div class="highlight"><pre><span></span><span class="nt">base</span><span class="p">:</span>
<span class="w"> </span><span class="s">'*'</span><span class="p p-Indicator">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt.mine</span>
</pre></div>
<p>刷新Pillar,并验证Salt Mine配置</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>saltutil.refresh_pillar
salt<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>pillar.get<span class="w"> </span>mine_functions
</pre></div>
<p>更新Salt Mine,并测试获取所有Minions的ID及IP</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>mine.update
salt<span class="w"> </span><span class="s1">'salt'</span><span class="w"> </span>mine.get<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>network.ip_addrs
</pre></div>
</div>
<div class="section" id="salt-rosters">
<h3>生成Salt Rosters</h3>
<p>配置Salt Rosters state</p>
<p><em>/srv/salt/salt/ssh/roster.sls</em></p>
<div class="highlight"><pre><span></span><span class="nt">salt-rosters</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">{</span><span class="c1"># salt rosters file for salt-ssh #}</span>
<span class="nt"> file.managed</span><span class="p">:</span>
<span class="nt"> - name</span><span class="p">:</span><span class="w"> </span><span class="nv">/etc/salt/roster</span>
<span class="w"> </span><span class="nv">- source</span><span class="p p-Indicator">:</span><span class="w"> </span><span class="nv">salt</span><span class="p p-Indicator">:</span><span class="nv">//salt/files/etc/salt/roster</span>
<span class="w"> </span><span class="nv">- template</span><span class="p p-Indicator">:</span><span class="w"> </span><span class="nv">jinja</span>
</pre></div>
<p><em>/srv/salt/salt/files/etc/salt/roster</em>:</p>
<pre class="literal-block">
{% for eachminion, each_mine in salt['mine.get']('*', 'network.ip_addrs').iteritems() -%}
{{eachminion}}:
host: {{each_mine[0]}}
user: salt
sudo: True
{% endfor -%}
</pre>
<p>生成Salt Rosters</p>
<div class="highlight"><pre><span></span>salt<span class="w"> </span><span class="s1">'salt'</span><span class="w"> </span>state.sls<span class="w"> </span>salt.ssh.roster
</pre></div>
</div>
<div class="section" id="salt-ssh">
<h3>应用Salt SSH</h3>
<p>将Master升级至0.17及以上版本(EPEL Stable当前版本为已经为0.17.1-1), 至此, Salt SSH环境已经构建完毕</p>
<div class="highlight"><pre><span></span>yum<span class="w"> </span>update<span class="w"> </span>salt-master
service<span class="w"> </span>salt-master<span class="w"> </span>restart
</pre></div>
<p>测试Salt SSH</p>
<div class="highlight"><pre><span></span><span class="c1">## 运行Salt Module</span>
salt-ssh<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>test.ping
<span class="c1">## 运行原始SHELL命令</span>
salt-ssh<span class="w"> </span><span class="s1">'*'</span><span class="w"> </span>-r<span class="w"> </span><span class="s1">'uptime'</span>
</pre></div>
</div>
</div>
<div class="section" id="section-4">
<h2>后话</h2>
<p>Salt这是要抢 <a class="reference external" href="https://github.com/fabric/fabric">Fabric</a> 饭碗的节奏啊,个人更喜欢Salt Master/Minions这样的管理方式,Salt SSH作为补充,用于升级Minion、重启Minion等等自维护工作还是很靠谱的。有了Salt SSH,再也不用担心是先有鸡还是先有蛋的问题了.</p>
</div>
【翻译】如何建立多Master的SaltStack环境2013-09-25T00:00:00+08:002013-09-25T00:00:00+08:00pengyaotag:pengyao.org,2013-09-25:/howto_configure_a_multi_master_saltstack_setup.html<p class="first last">翻译的《How To Configure A Multi-Master SaltStack Setup》</p>
<ul class="simple">
<li>英文原文出处: <a class="reference external" href="http://intothesaltmine.org/how_to_configure_a_multi_master_saltstack_setup.html">http://intothesaltmine.org/how_to_configure_a_multi_master_saltstack_setup.html</a></li>
</ul>
<p>0.16.0版本的发布,带来了minion可以连接多Master的特性. 这种方式称为多master( <tt class="docutils literal"><span class="pre">multi-master</span></tt> )配置, 使环境中的SaltStack冗余。在这种配置下,Salt Minions将连接所有配置的Salt Master. 本文将带你了解如何建立多Master的环境.</p>
<div class="section" id="master-keys">
<h2>Master Keys</h2>
<p>在建立多Master的配置中,主要的一点就是所有的Master使用同样的private key. 这些key将在Master第一次启动时自动生成。 因此在多Master环境建立时,需要从原始的(original) Master上拷贝其private key至第二个Master以提供它启动时自动生成的那个, 以此类推.</p>
<p>Master的private key存储在Master本地的 <tt class="docutils literal">pki_dir</tt> 目录下. 默认的目录是 <tt class="docutils literal">/etc/salt/pki/master/master.pem</tt> . 将该key拷贝到新增的master上. 需要注意的是,在拷贝的时候,需要确保新增的master上并没有minion连接进来.</p>
</div>
<div class="section" id="configure-minions">
<h2>Configure Minions</h2>
<p>当配置多Master时,Minion需要知道需要连接的每个Master的网络地址. 需要在Minion的配置文件中进行配置,默认的配置文件是 <tt class="docutils literal">/etc/salt/minion</tt> 。 找到 <tt class="docutils literal">master</tt> 配置项, 更新需要新增的Master.</p>
<p>下边是一个多Master的配置例子:</p>
<div class="highlight"><pre><span></span><span class="nt">master</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">master1.example.tld</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">master2.example.tld</span>
</pre></div>
<p>配置完毕后,需要重启Minion以确保配置生效. 此时所有的Master均可以控制你的minions.</p>
</div>
<div class="section" id="sharing-files-between-masters">
<h2>Sharing Files Between Masters</h2>
<p>Salt并不会自动在Master间共享文件. 本小节将带你了解Master间哪些文件需要同步以保持一致.</p>
<div class="section" id="minion-keys">
<h3>Minion Keys</h3>
<p>Minion的keys需要每个Master都进行accept. 可以使用 <tt class="docutils literal"><span class="pre">salt-key</span></tt> 手动接接受minion的key, 也可以在Master间保持key目录的同步. 需要同步的目录有:</p>
<ul class="simple">
<li>/etc/salt/pki/master/minions</li>
<li>/etc/salt/pki/master/minions_pre</li>
<li>/etc/salt/pki/master/minions_rejected</li>
</ul>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">直接共享 <tt class="docutils literal">/etc/salt/master</tt> 目录是强烈反对的. 允许外部访问 master.pem key将带来严重的安全风险</p>
</div>
</div>
<div class="section" id="file-roots">
<h3>file_roots</h3>
<p><tt class="docutils literal">file_roots</tt> 的内容需要在Master间同步以保持一致. 这里存放Salt State配置管理文件. 推荐同步内容使用 <tt class="docutils literal">gitfs</tt> backend,或者直接将 <tt class="docutils literal">file_roots</tt> 存储在共享存储上.</p>
</div>
<div class="section" id="pillar-roots">
<h3>pillar_roots</h3>
<p>同理,对于 <tt class="docutils literal">pillar_roots</tt> 也是如此,需要保持Pillar数据一致.</p>
</div>
<div class="section" id="master-configuration">
<h3>Master Configuration</h3>
<p>最后你需要确保有关Master的配置选项在所有Master间是同步的. 除非你知道你不需要这么做,你需要保证以下的设置Master间是同步的:</p>
<blockquote>
<ul class="simple">
<li>external_auth</li>
<li>client_acl</li>
<li>peer</li>
<li>peer_run</li>
</ul>
</blockquote>
</div>
</div>
<div class="section" id="conslusion">
<h2>Conslusion</h2>
<p>多Master环境配置提供了控制Minions的冗余性,配置相当简单. 只需要保证key及State文件在你的多Master间是同步的,你就可以透明的在多Master上控制你的Minions</p>
</div>
Salt Runners manage学习2013-09-24T00:00:00+08:002013-09-24T00:00:00+08:00pengyaotag:pengyao.org,2013-09-24:/salt_runners_manage_01.html<p class="first last">参考手册和源码对Salt Runners manage进行学习</p>
<ul class="simple">
<li>Salt Runners manage手册: <a class="reference external" href="http://docs.saltstack.com/ref/runners/all/salt.runners.manage.html?highlight=manage#salt.runners.manage">http://docs.saltstack.com/ref/runners/all/salt.runners.manage.html?highlight=manage#salt.runners.manage</a></li>
<li>Salt Runners manage源码: <a class="reference external" href="https://github.com/saltstack/salt/blob/develop/salt/runners/manage.py">https://github.com/saltstack/salt/blob/develop/salt/runners/manage.py</a></li>
</ul>
<p>下午灿哥在群里边分享了 <tt class="docutils literal"><span class="pre">salt-run</span> manage.status</tt> 的用法,用于检查minion当前是否存活(可连接). 这个功能果然不错,索性就打开对应的源码,对manage所有的方法进行一次学习</p>
<ul class="simple">
<li>版本: 0.16.3</li>
</ul>
<div class="section" id="status">
<h2>status</h2>
<ul class="simple">
<li>使用方法: <tt class="docutils literal"><span class="pre">salt-run</span> manage.status</tt></li>
<li>功能: 输出所有已知的minions的状态, 以up和down分组输出</li>
<li>核心代码及补充的代码说明:</li>
</ul>
<div class="highlight"><pre><span></span><span class="n">client</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">LocalClient</span><span class="p">(</span><span class="n">__opts__</span><span class="p">[</span><span class="s1">'conf_file'</span><span class="p">])</span>
<span class="n">minions</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">cmd</span><span class="p">(</span><span class="s1">'*'</span><span class="p">,</span> <span class="s1">'test.ping'</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="n">__opts__</span><span class="p">[</span><span class="s1">'timeout'</span><span class="p">])</span> <span class="c1">#利用client.cmd对所有的minion发送test.ping指令,用于探测minion是否存活</span>
<span class="n">key</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">key</span><span class="o">.</span><span class="n">Key</span><span class="p">(</span><span class="n">__opts__</span><span class="p">)</span>
<span class="n">keys</span> <span class="o">=</span> <span class="n">key</span><span class="o">.</span><span class="n">list_keys</span><span class="p">()</span> <span class="c1"># 利用salt.key获取当前master上有多少minion的key,即获取到完整的minion列表</span>
<span class="n">ret</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">ret</span><span class="p">[</span><span class="s1">'up'</span><span class="p">]</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">minions</span><span class="p">)</span> <span class="c1"># 将执行test.ping有返回值的minion即存活的minion的ID放入up中</span>
<span class="n">ret</span><span class="p">[</span><span class="s1">'down'</span><span class="p">]</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">(</span><span class="nb">set</span><span class="p">(</span><span class="n">keys</span><span class="p">[</span><span class="s1">'minions'</span><span class="p">])</span> <span class="o">-</span> <span class="nb">set</span><span class="p">(</span><span class="n">minions</span><span class="p">))</span> <span class="c1">#完整的minion列表减去存活的minion就是down掉/无法连接的minion喽</span>
<span class="k">if</span> <span class="n">output</span><span class="p">:</span>
<span class="n">salt</span><span class="o">.</span><span class="n">output</span><span class="o">.</span><span class="n">display_output</span><span class="p">(</span><span class="n">ret</span><span class="p">,</span> <span class="s1">''</span><span class="p">,</span> <span class="n">__opts__</span><span class="p">)</span> <span class="c1"># 输出</span>
<span class="k">return</span> <span class="n">ret</span>
</pre></div>
<ul class="simple">
<li>总结: 该方法果然很给力,从此妈妈再也不担心不知道minion是否存活喽</li>
</ul>
</div>
<div class="section" id="key-regen">
<h2>key_regen</h2>
<ul class="simple">
<li>使用方法: <tt class="docutils literal"><span class="pre">salt-run</span> manage.key_regen</tt></li>
<li>功能: 重新生成环境下的所有key (副作用甚强,慎用,慎用, 除非你知道在做什么)</li>
<li>核心代码及补充的代码说明:</li>
</ul>
<div class="highlight"><pre><span></span><span class="n">minions</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">cmd</span><span class="p">(</span><span class="s1">'*'</span><span class="p">,</span> <span class="s1">'saltutil.regen_keys'</span><span class="p">)</span> <span class="c1"># 执行saltutil.regen_keys,重新生成key</span>
</pre></div>
<ul class="simple">
<li>总结: 慎用,慎用,慎用</li>
</ul>
</div>
<div class="section" id="down">
<h2>down</h2>
<ul class="simple">
<li>使用方法: <tt class="docutils literal"><span class="pre">salt-run</span> manage.down</tt></li>
<li>功能: 输出down掉/无法连接的minion</li>
<li>核心代码及补充的代码说明:</li>
</ul>
<div class="highlight"><pre><span></span><span class="n">ret</span> <span class="o">=</span> <span class="n">status</span><span class="p">(</span><span class="n">output</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'down'</span><span class="p">,</span> <span class="p">[])</span> <span class="c1"># 直接用之前的status方法,然后获取down的minion列表</span>
</pre></div>
<ul class="simple">
<li>总结: 函数编程果然是王道,省时省力,直接通过该方法查询down掉的minion,再也不麻烦了</li>
</ul>
</div>
<div class="section" id="up">
<h2>up</h2>
<ul class="simple">
<li>使用方法: <tt class="docutils literal"><span class="pre">salt-run</span> manage.up</tt></li>
<li>功能: 输出存活的minion</li>
<li>核心代码及补充的代码说明:</li>
</ul>
<div class="highlight"><pre><span></span><span class="n">ret</span> <span class="o">=</span> <span class="n">status</span><span class="p">(</span><span class="n">output</span><span class="o">=</span><span class="kc">False</span><span class="p">)</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'up'</span><span class="p">,</span> <span class="p">[])</span> <span class="c1"># 和上边直接down一样,不过这次的需求变成了up而已</span>
</pre></div>
<ul class="simple">
<li>总结: 和楼上类似</li>
</ul>
</div>
<div class="section" id="versions">
<h2>versions</h2>
<ul class="simple">
<li>使用方法: <tt class="docutils literal"><span class="pre">salt-run</span> manage.versions</tt></li>
<li>功能: 输出所有存活的minion的版本和master的版本对比情况</li>
<li>核心代码及补充的代码说明:</li>
</ul>
<div class="highlight"><pre><span></span><span class="n">minions</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">cmd</span><span class="p">(</span><span class="s1">'*'</span><span class="p">,</span> <span class="s1">'test.version'</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="n">__opts__</span><span class="p">[</span><span class="s1">'timeout'</span><span class="p">])</span> <span class="c1"># 通过client.cmd方法下发所有minion需要执行test.version(输出版本号)的指令</span>
<span class="n">labels</span> <span class="o">=</span> <span class="p">{</span> <span class="c1"># 定义版本对比的描述</span>
<span class="o">-</span><span class="mi">1</span><span class="p">:</span> <span class="s1">'Minion requires update'</span><span class="p">,</span>
<span class="mi">0</span><span class="p">:</span> <span class="s1">'Up to date'</span><span class="p">,</span>
<span class="mi">1</span><span class="p">:</span> <span class="s1">'Minion newer than master'</span><span class="p">,</span>
<span class="p">}</span>
<span class="n">version_status</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">comps</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">__version__</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'-'</span><span class="p">)</span> <span class="c1"># 获取master version</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">comps</span><span class="p">)</span> <span class="o">==</span> <span class="mi">3</span><span class="p">:</span>
<span class="n">master_version</span> <span class="o">=</span> <span class="s1">'-'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">comps</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">2</span><span class="p">])</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">master_version</span> <span class="o">=</span> <span class="n">salt</span><span class="o">.</span><span class="n">__version__</span>
<span class="k">for</span> <span class="n">minion</span> <span class="ow">in</span> <span class="n">minions</span><span class="p">:</span>
<span class="n">comps</span> <span class="o">=</span> <span class="n">minions</span><span class="p">[</span><span class="n">minion</span><span class="p">]</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">'-'</span><span class="p">)</span>
<span class="k">if</span> <span class="nb">len</span><span class="p">(</span><span class="n">comps</span><span class="p">)</span> <span class="o">==</span> <span class="mi">3</span><span class="p">:</span>
<span class="n">minion_version</span> <span class="o">=</span> <span class="s1">'-'</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">comps</span><span class="p">[</span><span class="mi">0</span><span class="p">:</span><span class="mi">2</span><span class="p">])</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">minion_version</span> <span class="o">=</span> <span class="n">minions</span><span class="p">[</span><span class="n">minion</span><span class="p">]</span>
<span class="n">ver_diff</span> <span class="o">=</span> <span class="n">cmp</span><span class="p">(</span><span class="n">minion_version</span><span class="p">,</span> <span class="n">master_version</span><span class="p">)</span> <span class="c1"># 通过python的cmp方法对版本号进行对比</span>
<span class="k">if</span> <span class="n">ver_diff</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">version_status</span><span class="p">:</span>
<span class="n">version_status</span><span class="p">[</span><span class="n">ver_diff</span><span class="p">]</span> <span class="o">=</span> <span class="p">[]</span>
<span class="n">version_status</span><span class="p">[</span><span class="n">ver_diff</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">minion</span><span class="p">)</span>
<span class="n">ret</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">for</span> <span class="n">key</span> <span class="ow">in</span> <span class="n">version_status</span><span class="p">:</span>
<span class="k">for</span> <span class="n">minion</span> <span class="ow">in</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">version_status</span><span class="p">[</span><span class="n">key</span><span class="p">]):</span>
<span class="n">ret</span><span class="o">.</span><span class="n">setdefault</span><span class="p">(</span><span class="n">labels</span><span class="p">[</span><span class="n">key</span><span class="p">],</span> <span class="p">[])</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">minion</span><span class="p">)</span>
<span class="n">salt</span><span class="o">.</span><span class="n">output</span><span class="o">.</span><span class="n">display_output</span><span class="p">(</span><span class="n">ret</span><span class="p">,</span> <span class="s1">''</span><span class="p">,</span> <span class="n">__opts__</span><span class="p">)</span>
<span class="k">return</span> <span class="n">ret</span>
</pre></div>
<ul class="simple">
<li>总结: 利用本方法,哪些minion需要升级立马得知,谁用谁知道啊!</li>
</ul>
</div>
【翻译】在SaltStack中如何使用require及watch语法2013-09-18T00:00:00+08:002013-09-18T00:00:00+08:00pengyaotag:pengyao.org,2013-09-18:/howto_to_use_require_and_watch_statements.html<p class="first last">翻译了intothesaltmine的《HOW TO USE REQUIRE AND WATCH STATEMENTS》,本文主要讲述在SaltStack中,如何使用require及watch语法来确定state间的依赖关系及执行顺序.</p>
<ul class="simple">
<li>原文出处: <a class="reference external" href="http://intothesaltmine.org/how_to_use_require_and_watch_statements.html">http://intothesaltmine.org/how_to_use_require_and_watch_statements.html</a></li>
<li>译者: <a class="reference external" href="http://pengyao.org/">pengyao</a></li>
</ul>
<p>在SaltStack配置关系系统中支持许多强大的选项。无论是简单的如软件包的安装还是使用模板和条件语句. SaltStack States可以从小巧逐步变得很复杂。幸运的是SaltStack提供一种用于解决States间依赖关系的方法. 本小节将讲述如何使用 <tt class="docutils literal">require</tt>、<tt class="docutils literal">require_in</tt>、<tt class="docutils literal">watch</tt>、<tt class="docutils literal">watch_in</tt></p>
<div class="section" id="requisites">
<h2>Requisites</h2>
<p>在SaltStack的世界中,requisites(译者注: 该词没找到合适的中文翻译,暂时使用英文原词)有两种类型,直接的requisites和"requisite_ins"。这些requisites是方向性的(directional),用于指定说"我依赖于某些东西"或"一些东西依赖于我"</p>
<div class="section" id="require">
<h3>require</h3>
<p>下边是使用 <tt class="docutils literal">require</tt> 语法的例子:</p>
<div class="highlight"><pre><span></span><span class="nt">vim</span><span class="p">:</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">pkg.installed</span>
<span class="nt">/etc/vimrc</span><span class="p">:</span>
<span class="w"> </span><span class="nt">file.managed</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">source</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt://edit/vimrc</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">pkg</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">vim</span>
</pre></div>
<p>在这个例子中, <tt class="docutils literal">/etc/vimrc</tt> 文件并不会被placed(managed),直到 <tt class="docutils literal">vim</tt> 软件包已安装</p>
</div>
<div class="section" id="require-in">
<h3>require_in</h3>
<p>下边是同样的例子,只是这次使用了 <tt class="docutils literal">require_in</tt> :</p>
<div class="highlight"><pre><span></span><span class="nt">vim</span><span class="p">:</span>
<span class="w"> </span><span class="nt">pkg.installed</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">require_in</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/etc/vimrc</span>
<span class="nt">/etc/vimrc</span><span class="p">:</span>
<span class="w"> </span><span class="nt">file.managed</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">source</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt://edit/vimrc</span>
</pre></div>
<p>这个例子的效果是相同的,在 <tt class="docutils literal">vim</tt> 中指定了 <tt class="docutils literal">/etc/vimrc</tt> 依赖于我</p>
<p>在最后,将会创建一个从属(dependency)map,并以有限的(finite)及可预见的(predictable)顺序执行.</p>
</div>
<div class="section" id="watch">
<h3>watch</h3>
<p>下面将以 <tt class="docutils literal">watch</tt> 语法举例,在本例中,运行中的 <tt class="docutils literal">ntpd</tt> 服务将会关注 <tt class="docutils literal">/etc/ntp.conf</tt> 文件的变化,如果发生变化,将会触发重启服务的操作.</p>
<div class="highlight"><pre><span></span><span class="nt">ntpd</span><span class="p">:</span>
<span class="w"> </span><span class="nt">service.running</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">watch</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">file</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">/etc/nto.conf</span>
<span class="nt">/etc/ntp.conf</span><span class="p">:</span>
<span class="w"> </span><span class="nt">file.managed</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">source</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt://ntp/files/ntp.conf</span>
</pre></div>
</div>
<div class="section" id="watch-in">
<h3>watch_in</h3>
<p>在接下来例子中, <tt class="docutils literal">/etc/ntp.conf</tt> 声明(declaring)它应该被 <tt class="docutils literal">ntpd</tt> 服务watch</p>
<div class="highlight"><pre><span></span><span class="nt">ntpd</span><span class="p">:</span>
<span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">service.running</span>
<span class="nt">/etc/ntp.conf</span><span class="p">:</span>
<span class="w"> </span><span class="nt">file.managed</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">source</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">salt://ntp/files/ntp.conf</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">watch_in</span><span class="p">:</span>
<span class="w"> </span><span class="p p-Indicator">-</span><span class="w"> </span><span class="nt">service</span><span class="p">:</span><span class="w"> </span><span class="l l-Scalar l-Scalar-Plain">ntpd</span>
</pre></div>
</div>
</div>
<div class="section" id="conclusion">
<h2>总结(Conclusion)</h2>
<p>在State规则中,你可以通过强大的 <tt class="docutils literal">require</tt> 、 <tt class="docutils literal">require_in</tt> 、 <tt class="docutils literal">watch</tt> 及 <tt class="docutils literal">watch_in</tt> 指定state间的依赖关系. 无论是一个服务应该watch一个文件的变化,还是一个服务运行前必须确保软件包已安装都可以通过它们来指定state的逻辑执行顺序.</p>
</div>
常见WebServer日志格式中请求处理时间汇总2013-05-15T00:00:00+08:002013-05-15T00:00:00+08:00pengyaotag:pengyao.org,2013-05-15:/webserver-request-time-1.html<p class="first last">不同WebServer的日志格式中对于请求处理时间使用的关键字及单位是不相同的,对常用的WebServer进行汇总.</p>
<p>如题,直接上表格:</p>
<table border="1" class="docutils">
<colgroup>
<col width="25%" />
<col width="33%" />
<col width="42%" />
</colgroup>
<thead valign="bottom">
<tr><th class="head">WebServer</th>
<th class="head">日志格式中对应的关键字</th>
<th class="head">单位</th>
</tr>
</thead>
<tbody valign="top">
<tr><td><a class="reference external" href="http://nginx.org">Nginx</a></td>
<td>$request_time</td>
<td>秒,小数点后可以精确到毫秒</td>
</tr>
<tr><td><a class="reference external" href="http://httpd.apache.org">Apache</a></td>
<td>%D 或 %T</td>
<td>%D为微秒, %T为秒</td>
</tr>
<tr><td><a class="reference external" href="http://tomcat.apache.org">Tomcat</a></td>
<td>%D 或 %T</td>
<td>%D为毫秒, %T为秒</td>
</tr>
<tr><td><a class="reference external" href="http://www.caucho.com/resin">Resin</a></td>
<td>%D 或 %T</td>
<td>%D为微秒, %T为秒</td>
</tr>
<tr><td><a class="reference external" href="http://www.iis.net">IIS</a></td>
<td>time-taken</td>
<td>微秒</td>
</tr>
</tbody>
</table>
reStructuredText 入门1989-06-04T00:00:00+08:001989-06-04T00:00:00+08:00pengyaotag:pengyao.org,1989-06-04:/rest-primer-chinese.html<p class="first last">转载的reStructuredText入门教程</p>
<p>原文出处: <a class="reference external" href="http://sphinx-doc-zh.readthedocs.org/en/latest/rest.htm">http://sphinx-doc-zh.readthedocs.org/en/latest/rest.htm</a></p>
<p>本节简要介绍了 新结构化文本 ~ reStructuredText的 (reST)的概念和语法,
旨在提供足够的信息来帮助作者高效起草文件.
由于 reST 被设计成一个简单的,不显眼的标记语言,
所以,这不会花太长时间.</p>
<div class="admonition note">
<p class="first admonition-title">Note</p>
<p class="last">权威 <a class="reference external" href="http://docutils.sourceforge.net/rst.html">新结构化文本用户文档</a>
在文章的 "ref" 链接中,有reST 各种结构的描述可供参考.</p>
</div>
<div class="section" id="paragraphs">
<h2>段落 Paragraphs</h2>
<p>段落( :<cite>参考 <paragraphs></cite> )是 reST 文章中最常见的文本块.
段落是由一个或多个空白分隔的文本块.
同Python中的约定,在 reST 中使用缩进来标识,
因此, 所有同级段落,必须左对齐,使用同级缩进.</p>
</div>
<div class="section" id="inline-markup">
<span id="inlinemarkup"></span><h2>行内标记 Inline markup</h2>
<p>标准的reST 行内标记很简单:</p>
<ul class="simple">
<li>单星号: <tt class="docutils literal">*文本*</tt> 得 <em>强调</em> (斜体 <sup>对中文一般效果不好</sup>) ,</li>
<li>双星号: <tt class="docutils literal">**文本**</tt> 得 <strong>加重</strong> (加黑),</li>
<li>反引号: <tt class="docutils literal">``文本``</tt> 得 代码引用.</li>
</ul>
<p>If asterisks or backquotes appear in running text and could be confused with
inline markup delimiters, they have to be escaped with a backslash.</p>
<p>如果有星号或反引号出现在引用的文本,
就可能会弄乱内联标记分隔符,这时,可以用反斜杠来转义.</p>
<p>Be aware of some restrictions of this markup:
以下是知道这些标记的一些限制:</p>
<ul class="simple">
<li>不可叠用</li>
<li>前后不能用空格: <tt class="docutils literal">* text*</tt> 这样会出错,</li>
<li>必须和周围文字用非单词隔离, 一般使用转义空白来完成: <tt class="docutils literal">thisis\ *one*\ word</tt></li>
</ul>
<p>These restrictions may be lifted in future versions of the docutils.
docutils未来版本中,可能取消这些限制.</p>
<p>reST also allows for custom "interpreted text roles"', which signify that the
enclosed text should be interpreted in a specific way. Sphinx uses this to
provide semantic markup and cross-referencing of identifiers, as described in
the appropriate section. The general syntax is <tt class="docutils literal"><span class="pre">:rolename:`content`</span></tt>.</p>
<p>reST 也支持自定"文本诠释规则",
这意味着,任意由指定字符封闭的文本都可以用特定的方式来诠释.
Sphinx 就用这种形式来提供语义标记和交叉引用,
一般语法形如: <tt class="docutils literal"><span class="pre">:规则名:`内容`</span></tt></p>
<p>Standard reST provides the following roles:
标准 reST 提供以下规则:</p>
<ul class="simple">
<li><cite>emphasis</cite> -- <tt class="docutils literal">*emphasis*</tt> 的替代拼写</li>
<li><cite>strong</cite> -- <tt class="docutils literal">**strong**</tt> 的替代拼写</li>
<li><cite>literal</cite> -- <tt class="docutils literal">``literal``</tt> 的替代拼写</li>
<li><cite>subscript</cite> -- 下标</li>
<li><cite>superscript</cite> -- 上标</li>
<li><cite>title-reference</cite> -- 书籍/期刊/及其他材料的标题</li>
</ul>
<p>参考: <cite>inline-markup</cite> Sphinx 追加的规则</p>
</div>
<div class="section" id="lists-and-quote-like-blocks">
<h2>列表和引用块 Lists and Quote-like blocks</h2>
<p>List markup (:<cite>ref <bullet-lists></cite>) is natural: just place an asterisk at
the start of a paragraph and indent properly. The same goes for numbered lists;
they can also be autonumbered using a <tt class="docutils literal">#</tt> sign::
列表标记(:<cite>参考 <bullet-lists></cite>): 只要自然的在段落的开始放置一个星号并正确缩进.
这同样适用于带编号的列表;
也可以使用``#``签署自动编号:</p>
<pre class="literal-block">
* This is a bulleted list.
* It has two items, the second
item uses two lines.
1. This is a numbered list.
2. It has two items too.
#. This is a numbered list.
#. It has two items too.
</pre>
<p>Nested lists are possible, but be aware that they must be separated from the
parent list items by blank lines</p>
<p>嵌套的列表是允许的但必须用空行同父列表分离开:</p>
<pre class="literal-block">
* this is
* a list
* with a nested list
* and some subitems
* and here the parent list continues
</pre>
<p>定义列表(:<cite>参考 <definition-lists></cite>) 如下创建:</p>
<pre class="literal-block">
term (up to a line of text)
Definition of the term, which must be indented
and can even consist of multiple paragraphs
next term
Description.
</pre>
<p>Note that the term cannot have more than one line of text.
注意, 条目本身不能多行.</p>
<p>Quoted paragraphs (:<cite>ref <block-quotes></cite>) are created by just indenting
them more than the surrounding paragraphs.
创建引用段落 (:<cite>参考 <block-quotes></cite>)只需要用缩进和其它段落区分即可.</p>
<p>线块 (:<cite>ref <line-blocks></cite>) 是保留换行符的一种方法:</p>
<pre class="literal-block">
| These lines are
| broken exactly like in
| the source file.
</pre>
<p>还有其它特殊文本块形式是支持的:</p>
<ul class="simple">
<li>字段列表 (field lists :<cite>参考 <field-lists></cite>)</li>
<li>选项列表 (option lists :<cite>参考 <option-lists></cite>)</li>
<li>引述文本块 (quoted literal blocks :<cite>参考 <quoted-literal-blocks></cite>)</li>
<li>文本测试块 (doctest blocks :<cite>参考 <doctest-blocks></cite>)</li>
</ul>
</div>
<div class="section" id="source-code">
<h2>源代码 Source Code</h2>
<p>代码文本块 (:<cite>参考 <literal-blocks></cite>) 由末尾是特殊标记 <tt class="docutils literal">::</tt> 的段落引发.
整个代码文本块必须缩进
(同所有的段落一样,使用空白行和周围文本完成分隔):</p>
<pre class="literal-block">
This is a normal text paragraph. The next paragraph is a code sample::
It is not processed in any way, except
that the indentation is removed.
It can span multiple lines.
This is a normal text paragraph again.
</pre>
<p><tt class="docutils literal">::</tt> 标记是智能处置的:</p>
<ul class="simple">
<li>如果作为一个独立段落出现,则和其它文本完全隔离</li>
<li>如果它紧跟有空格,则将被删除不起作用</li>
<li>如果它在非空白字符之前,则替换为普通的单一冒号</li>
</ul>
<p>综上,前述示例中的第二段代码引用文本之前的一句会渲染为 "The next paragraph is a code sample:"</p>
<p>That way, the second sentence in the above example's first paragraph would be
rendered as "The next paragraph is a code sample:".</p>
</div>
<div class="section" id="tables">
<span id="rst-tables"></span><h2>表格 Tables</h2>
<p>支持两种表格.</p>
<p><strong>网格表</strong> (:<cite>参考 <grid-tables></cite>),
你不得不自行"绘制"表格的边框.看起来象这样:</p>
<pre class="literal-block">
+------------------------+------------+----------+----------+
| Header row, column 1 | Header 2 | Header 3 | Header 4 |
| (header rows optional) | | | |
+========================+============+==========+==========+
| body row 1, column 1 | column 2 | column 3 | column 4 |
+------------------------+------------+----------+----------+
| body row 2 | ... | ... | |
+------------------------+------------+----------+----------+
</pre>
<p><strong>简单表</strong> (:<cite>参考 <simple-tables></cite>) 容易点,</p>
<p>但是有限制:至少要有一列,而且,第一行不能包含多行文本,
看起来象这样:</p>
<pre class="literal-block">
===== ===== =======
A B A and B
===== ===== =======
False False False
True False False
False True False
True True True
===== ===== =======
</pre>
</div>
<div class="section" id="hyperlinks">
<h2>超链接 Hyperlinks</h2>
<div class="section" id="external-links">
<h3>外部链接 External links</h3>
<p>Use <tt class="docutils literal">`Link text <span class="pre"><http://example.com/>`_</span></tt> for inline web links. If the link
text should be the web address, you don't need special markup at all, the parser
finds links and mail addresses in ordinary text.</p>
<p>用 <tt class="docutils literal">`Link text <span class="pre"><http://example.com/>`_</span></tt> 来记录行内链接.
如果文字本身就是链接,
那不用作任何标记,解析器可以自动将链接和邮箱地址转换为超链接.</p>
<p>也可以单独定义链接目标用引用(:<cite>参考 <hyperlink-targets></cite>),比如:</p>
<pre class="literal-block">
This is a paragraph that contains `a link`_.
.. _a link: http://example.com/
</pre>
</div>
<div class="section" id="internal-links">
<h3>内部链接 Internal links</h3>
<p>Internal linking is done via a special reST role provided by Sphinx, see the
section on specific markup, <cite>ref-role</cite>.</p>
<p>Sphinx 使用特殊 reST 规则支持内部链接,
详细参考 <cite>定义规则 <Mref-role></cite></p>
</div>
</div>
<div class="section" id="sections">
<h2>章节 Sections</h2>
<p>Section headers (:<cite>ref <sections></cite>) are created by underlining (and
optionally overlining) the section title with a punctuation character, at least
as long as the text</p>
<p>章节头部 (:<cite>参考 <sections></cite>)
由下线(也可有上线)和包含标点的标题 组合创建,
其中下线要至少等于标准文本的长度:</p>
<pre class="literal-block">
=================
This is a heading
=================
</pre>
<div class="sidebar">
<p class="first sidebar-title">注意</p>
<p class="sidebar-subtitle">中文标题的问题</p>
<p class="last">在多数编辑器中,全角/半角中文/标点和E文字符的长度是完全没谱的,
所以,多数情况下,为保持一致性,译者建议统一使用固定长度的上下标线;
比如说78.</p>
</div>
<p>Normally, there are no heading levels assigned to certain characters as the
structure is determined from the succession of headings. However, for the
Python documentation, this convention is used which you may follow:
通常并没有对标题的层级指定明确的标线字符.
不过,对于 Pyhton 文档,可以使用如下约定:</p>
<ul class="simple">
<li><tt class="docutils literal">#</tt> 有上标线, 用以部分</li>
<li><tt class="docutils literal">*</tt> 有上标线, 用以章节</li>
<li><tt class="docutils literal">=</tt>, 用以小节</li>
<li><tt class="docutils literal">-</tt>, 用以子节</li>
<li><tt class="docutils literal">^</tt>, 用以子节的子节</li>
<li><tt class="docutils literal">"</tt>, 用以段落</li>
</ul>
<p>Of course, you are free to use your own marker characters (see the reST
documentation), and use a deeper nesting level, but keep in mind that most
target formats (HTML, LaTeX) have a limited supported nesting depth.</p>
<p>当然,你可以自由的使用你自定的标识字符(参考 reST 文档),
并使用更加深的嵌套层次,
不过,考虑到兼容多种输出格式(HTML, LaTeX) 最好限制嵌套的深度.</p>
<div class="sidebar">
<p class="first sidebar-title">提示</p>
<p class="sidebar-subtitle">标题层次体验</p>
<p class="last">从行文来说,结构化文本组织的文章,更加关注局部文本的结构清晰,
以整个图书来说,不建议设定太多的标题级别,一般而言**四级**足够了.</p>
</div>
</div>
<div class="section" id="explicit-markup">
<h2>直解标记 Explicit Markup</h2>
<p>"Explicit markup" (:<cite>ref <explicit-markup-blocks></cite>) is used in reST for
most constructs that need special handling, such as footnotes,
specially-highlighted paragraphs, comments, and generic directives.</p>
<p>"直解标记" (Explicit markup, :<cite>参考 <explicit-markup-blocks></cite>)
用以 reST 中需要特殊处理的各种内容,
如脚注,特殊高亮段落,注释,以及通用指令.</p>
<p>An explicit markup block begins with a line starting with <tt class="docutils literal">..</tt> followed by
whitespace and is terminated by the next paragraph at the same level of
indentation. (There needs to be a blank line between explicit markup and normal
paragraphs. This may all sound a bit complicated, but it is intuitive enough
when you write it.)</p>
<p>直解标记块由``..``开始,紧后跟空格以及跟随的同缩进的文本块.
(和正文间要有空白行来明确的加以区分.
可能听起来有点复杂,但当你书写时就能直观的体验到)</p>
</div>
<div class="section" id="directives-1">
<span id="directives"></span><h2>指令 Directives</h2>
<p>A directive (:<cite>ref <directives></cite>) is a generic block of explicit markup.
指令(:<cite>ref <directives></cite>)就是一个标准的明确标记(Explicit Markup)块.
Besides roles, it is one of the extension mechanisms of reST, and Sphinx makes
heavy use of it.
除了规则,它是reST 的又一个扩展机制,
Sphinx 大量使用了指令.</p>
<p>Docutils 支持以下指令:</p>
<ul>
<li><p class="first">警示 Admonitions: <cite>attention</cite>, <cite>caution</cite>, <cite>danger</cite>,
<cite>error</cite>, <cite>hint</cite>, <cite>important</cite>, <cite>note</cite>,
<cite>tip</cite>, <cite>warning</cite> and the generic <cite>admonition</cite>.
(多数样式目前仅支持 "note" 和 "warning" <sup>好在都有针对的对象ID,很容易使用CSS进行定制</sup> .)</p>
</li>
<li><p class="first">图像 Images:</p>
<ul class="simple">
<li><cite>image</cite></li>
<li><cite>figure</cite> (配有标题和图例 的图片)</li>
</ul>
</li>
<li><p class="first">其它行文元素 Additional body elements:</p>
<ul class="simple">
<li><cite>contents</cite> (对诸如 本地文件 的内容表单)</li>
<li><cite>container</cite> (配有定制 class 的容器,以便生成HTML 中的 <tt class="docutils literal"><div></tt> )</li>
<li><cite>rubric</cite> (没有到相对段落关系的标题 a heading without relation to the document sectioning)</li>
<li><cite>topic</cite>, <cite>sidebar</cite> (特殊高亮的正文元素 special highlighted body elements)</li>
<li><cite>parsed-literal</cite> (支持内嵌标记的文本块)</li>
<li><cite>epigraph</cite> (有可选归属行的引用文本块)</li>
<li><cite>highlights</cite>, <cite>pull-quote</cite> (有他们自己class属性的文本块)</li>
<li><cite>compound</cite> (复合段落)</li>
</ul>
</li>
<li><p class="first">特殊表格 Special tables:</p>
<ul class="simple">
<li><cite>table</cite> (有标题的表格)</li>
<li><cite>csv-table</cite> (从csv数据生成的表格)</li>
<li><cite>list-table</cite> (从列表数据生成的表格)</li>
</ul>
</li>
<li><p class="first">特殊指令 Special directives:</p>
<ul class="simple">
<li><cite>raw</cite> (包括原始文本的目标格式标记 include raw target-format markup)</li>
<li><cite>include</cite> (从其它文件引入 reST )
-- 在Sphinx, 当给定包含文件的绝对路径时,指令会从源代码目录为起点进行相对路径查找.</li>
<li><cite>class</cite> (将 class 属性绑定到下一个元素) <a class="footnote-reference" href="#footnote-1" id="footnote-reference-1">[1]</a></li>
</ul>
</li>
<li><p class="first">HTML 专用 specifics:</p>
<ul class="simple">
<li><cite>meta</cite> (生成 HTML 中的 <tt class="docutils literal"><meta></tt> 标签)</li>
<li><cite>title</cite> (覆盖文件标题)</li>
</ul>
</li>
<li><p class="first">影响标记 Influencing markup:</p>
<ul class="simple">
<li><cite>default-role</cite> (设置新默认规则)</li>
<li><cite>role</cite> (创建新规则)</li>
</ul>
<p>由于这些指令都只能作用到单一文件,所以,更好的使用 Sphinx 的方式是设置 <cite>default_role</cite>.</p>
</li>
</ul>
<p><em>不要</em> 使用指令 <cite>sectnum</cite>, <cite>header</cite> 和 <cite>footer</cite>.</p>
<p>Sphinx 增加的指令描述收集在: <cite>sphinxmarkup</cite> .</p>
<p>Basically, a directive consists of a name, arguments, options and content. (Keep
this terminology in mind, it is used in the next chapter describing custom
directives.) Looking at this example,
基本上一个指令由名称,参数,选项和内容组成.
(请记住这里提及的几个术语,
它们将在之后章节描述自定义指令)
从这个例子来看,:</p>
<pre class="literal-block">
.. function:: foo(x)
foo(y, z)
:module: some.module.name
Return a line of text input from the user.
</pre>
<p><tt class="docutils literal">function</tt> 是指令名,
在头两行里给出了两个参数,
紧接着给出了一个 <tt class="docutils literal">module</tt> 选项
(正如你所见,由冒号标明的 <tt class="docutils literal">module</tt> 之后立即跟上参数)
选项必须缩进和指令内容有相同的缩进.</p>
<!-- The directive content follows after a blank line and is indented relative to the directive start. -->
<p>该指令的内容则是由一个空行和同样的缩进来接上.</p>
</div>
<div class="section" id="images">
<h2>图片 Images</h2>
<p>reST 支持图片指令 (<cite>ref <image></cite>), 这样使用:</p>
<pre class="literal-block">
.. image:: gnu.png
(options)
</pre>
<p>在Sphinx 中使用时,
给入的文件名 (此处是 <tt class="docutils literal">gnu.png</tt>) 必须是相对源文件目录的路径,
如果给的是绝对路径形式,也意味着对源文件顶层目录进行相对查找.
比如说, 文件 <tt class="docutils literal">sketch/spam.rst</tt> 可以用路径 <tt class="docutils literal"><span class="pre">../images/spam.png</span></tt> 或 <tt class="docutils literal">/images/spam.png</tt>.
来引用图片 <tt class="docutils literal">images/spam.png</tt></p>
<p>Sphinx will automatically copy image files over to a subdirectory of the output
directory on building (e.g. the <tt class="docutils literal">_static</tt> directory for HTML output.)
Sphinx 会自动将图片复制到构筑输出目录中的相关子目录
(e.g. HTML输出时的 <tt class="docutils literal">_static</tt> 目录.)</p>
<p>Interpretation of image size options (<tt class="docutils literal">width</tt> and <tt class="docutils literal">height</tt>) is as follows:
if the size has no unit or the unit is pixels, the given size will only be
respected for output channels that support pixels (i.e. not in LaTeX output).
Other units (like <tt class="docutils literal">pt</tt> for points) will be used for HTML and LaTeX output.</p>
<p>图片尺寸的解释选项 (<tt class="docutils literal">width</tt> 和 <tt class="docutils literal">height</tt>)有如下规约:
如果大小没给任何单位或单位是像素,
输出通道优先使用像素(换言之,非LaTeX输出).
其他单位(如 <tt class="docutils literal">pt</tt> 或是 点) 将被用于HTML和LaTeX输出.</p>
<p>Sphinx extends the standard docutils behavior by allowing an asterisk for the
extension
Sphinx 扩展了标准 docutils 行为,支持如下的星号指代:</p>
<pre class="literal-block">
.. image:: gnu.*
</pre>
<p>Sphinx then searches for all images matching the provided pattern and determines
their type. Each builder then chooses the best image out of these candidates.
For instance, if the file name <tt class="docutils literal">gnu.*</tt> was given and two files <cite>gnu.pdf</cite>
and <cite>gnu.png</cite> existed in the source tree, the LaTeX builder would choose
the former, while the HTML builder would prefer the latter.
Sphinx 会搜索所有匹配所提供模式的图片,
并确定它们的类型.
每个构筑器再从中选择最佳的图片.
例如,
如果给定文件名是 <tt class="docutils literal">gnu.*</tt> ,
源代码树中有两个文件 <cite>gnu.pdf</cite> 和 <cite>gnu.png</cite> ,
LaTeX 构筑器会选择前者,
HTML 构筑器更倾向于后者.</p>
<!-- :: 0.4
增加了文件名的星号后缀支持. -->
<!-- :: 0.6
开始支持绝对路径的图片 -->
</div>
<div class="section" id="footnotes">
<h2>脚注 Footnotes</h2>
<p>and add the footnote body at the bottom of the document after a
"Footnotes" rubric heading, like so::
脚注 (:<cite>参考 <footnotes></cite>), 使用 <tt class="docutils literal">[#name]_</tt> 来标记位置,
并在文章底部 "Footnotes" 专栏之后追加脚注内容,如下使用:</p>
<pre class="literal-block">
Lorem ipsum [#f1]_ dolor sit amet ... [#f2]_
.. rubric:: Footnotes
.. [#f1] Text of the first footnote.
.. [#f2] Text of the second footnote.
</pre>
<p>You can also explicitly number the footnotes (<tt class="docutils literal">[1]_</tt>) or use auto-numbered
footnotes without names (<tt class="docutils literal"><span class="pre">[#]_</span></tt>).
可以使用确切编号的脚注 (如: <tt class="docutils literal">[1]_</tt>)
或是自动编号(用 <tt class="docutils literal"><span class="pre">[#]_</span></tt>).</p>
</div>
<div class="section" id="citations">
<h2>引证 Citations</h2>
<p>标准 reST 支持引证 (:<cite>参考 <citations></cite>) ,
with the
additional feature that they are "global", i.e. all citations can be referenced
from all files. Use them like so::
有额外的功能是 "global",
换言之,引证能从所有文件来引用.
这样使用:</p>
<pre class="literal-block">
Lorem ipsum [Ref]_ dolor sit amet.
.. [Ref] Book or article reference, URL or whatever.
</pre>
<p>Citation usage is similar to footnote usage, but with a label that is not
numeric or begins with <tt class="docutils literal">#</tt>.
引证 的使用基本和脚注相同,
不过使用的标签不是数字或是以 <tt class="docutils literal">#</tt> 开始.</p>
</div>
<div class="section" id="substitutions">
<h2>替换 Substitutions</h2>
<p>reST 支持 "替换" (:<cite>参考 <substitution-definitions></cite>),
以 <tt class="docutils literal">|name|</tt> 形式来定义替换的文本或是标记对象.
如脚注,可以在直解标记文本块中声明,形如:</p>
<pre class="literal-block">
.. |name| replace:: replacement *text*
</pre>
<p>或是:</p>
<pre class="literal-block">
.. |caution| image:: warning.png
:alt: Warning!
</pre>
<p>详参 :<cite>reST 替换参考 <substitution-definitions></cite> .</p>
<p>If you want to use some substitutions for all documents, put them into
<cite>rst_prolog</cite> or put them into a separate file and include it into all
documents you want to use them in, using the <cite>include</cite> directive. (Be
sure to give the include file a file name extension differing from that of other
source files, to avoid Sphinx finding it as a standalone document.)</p>
<p>如果你对所有文件使用一组替换,
把它们置入 <cite>rst_prolog</cite> 或放入一个单独的文件,
并在所有相关文件中使用 <cite>incluse</cite> 指令引入,
(请将此定义文件,使用和内容文件不同的后缀,否则,Sphinx 将视其为独立文章来尝试解析)</p>
<p>Sphinx defines some default substitutions, see <cite>default-substitutions</cite>.
Sphinx 本身有些默认替换,参考 <cite>default-substitutions</cite> .</p>
</div>
<div class="section" id="comments">
<h2>注释 Comments</h2>
<p>所有直解标记文本块都不算有效的标记构成
Every explicit markup block which isn't a valid markup construct (like the
footnotes above) is regarded as a comment (:<cite>ref <comments></cite>). For
example</p>
<p>没有有效标记(如脚注)的直解标记文本块就是注释(:<cite>参考 <comments></cite>)
例如:</p>
<pre class="literal-block">
.. This is a comment.
</pre>
<p>可以用缩进文本来进行多行注释:</p>
<pre class="literal-block">
..
This whole indented block
is a comment.
Still in the comment.
</pre>
</div>
<div class="section" id="source-encoding">
<h2>源文本编码 Source encoding</h2>
<p>Since the easiest way to include special characters like em dashes or copyright
signs in reST is to directly write them as Unicode characters, one has to
specify an encoding. Sphinx assumes source files to be encoded in UTF-8 by
default; you can change this with the <cite>source_encoding</cite> config value.</p>
<p>由于最简单的方式,是在 reST 中将包括特殊字符(如长划线或版权标记)都直接写成Unicode字符.
Sphinx 默认假设源文件是 utf-8 编码.
你可以用配置项 <cite>source_encoding</cite> 来指定别的编码.</p>
</div>
<div class="section" id="gotchas">
<h2>嗯嗯嗯 Gotchas</h2>
<p>There are some problems one commonly runs into while authoring reST documents:
通常运用 reST 进行撰写时会遇见几个问题:</p>
<ul class="simple">
<li><strong>对在线标记的分隔:</strong> 如前所述,内联标记必须用非单词字符和周围的文字进行区隔,
要解决这个问题你必须使用反斜杠转义空格,详见 <a class="reference external" href="http://docutils.sf.net/docs/ref/rst/restructuredtext.html#inline-markup">参考</a> .</li>
<li><strong>在线标记不能嵌套:</strong> 但是形如 <tt class="docutils literal">*see <span class="pre">:func:`foo`*</span></tt> 是没问题的.</li>
</ul>
<p class="rubric">Footnotes</p>
<table class="docutils footnote" frame="void" id="footnote-1" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#footnote-reference-1">[1]</a></td><td>当默认域包含 <cite>class</cite> 指令时,该指令将被掩蔽,
因此 Sphinx 转而使用 <cite>rst-class</cite>.</td></tr>
</tbody>
</table>
</div>