Florent Flament - OpenStackhttp://www.florentflament.com/blog/2014-10-20T23:25:00+02:00Splitting Swift cluster2014-10-20T23:25:00+02:002014-10-20T23:25:00+02:00Florent Flamenttag:www.florentflament.com,2014-10-20:/blog/splitting-swift-cluster.html<p>At <a href="https://www.cloudwatt.com/en/">Cloudwatt</a>, we have been operating a near hundred nodes Swift
cluster in a unique datacenter for a few years. The decision to split
the cluster on two datacenters has been taken recently. The goal is to
have at least one replica of each object on each site in order …</p><p>At <a href="https://www.cloudwatt.com/en/">Cloudwatt</a>, we have been operating a near hundred nodes Swift
cluster in a unique datacenter for a few years. The decision to split
the cluster on two datacenters has been taken recently. The goal is to
have at least one replica of each object on each site in order to
avoid data loss in case of the destruction of a full datacenter (fire,
plane crash, ...).</p>
<h2>Constraints when updating a running cluster</h2>
<p>Some precautions have to be taken when updating a running cluster with
customers' data. We want to ensure that no data is lost or corrupted
during the operation and that the cluster's performance isn't hurt too
badly. </p>
<p>In order to ensure that no data is lost, we have to follow some
guidelines including:</p>
<ul>
<li>Never move more that 1 replica of any object at any given step; That
way we ensure that 2 copies out 3 are left intact in case something
goes wrong.</li>
<li>Process by small steps to limit the impact in case of failure.</li>
<li>Check during each step that there is no unusual data corruptions,
and that corrupted data are correctly handled and fixed.</li>
<li>Check after each step that data has been moved (or kept) at the
correct place.</li>
<li>If any issue were to happen, rollback to previous step.</li>
</ul>
<p>To limit the impact on cluster's performance, we have to address to
following issues:</p>
<ul>
<li>Assess the availability of cluster resources (network bandwidth,
storage nodes' disks & CPU availability) at different times of day
and week. This would allow to choose the best time to perform our
steps.</li>
<li>Assess the load on the cluster of the steps planned to split the
cluster.</li>
<li>Choose steps small enough so that:<ul>
<li>it fits time frames where cluster's resources are more available;</li>
<li>the load incurred by the cluster (and its users) is acceptable.</li>
</ul>
</li>
</ul>
<p>A number of these requirements have been addressed by Swift for a while:</p>
<ul>
<li>When updating Swift ring files, the <code>swift-ring-builder</code> tool
doesn't move more than 1 replica during reassignment of cluster's
partitions (unless something really went wrong). By performing only
one reassignment per process step, we ensure that we don't move more
than 1 replica at each step.</li>
<li>Checking for data corruption is made easy by Swift. 3 processes
(<code>swift-object-auditor</code>, <code>swift-container-auditor</code> and
<code>swift-account-auditor</code>) running on storage nodes are continuously
checking and fixing data integrity.</li>
<li>Checking that data is at the correct location is also made easy by
the <code>swift-dispersion-report</code> provided.</li>
<li>Updating the location of data is made seamless by updating and
copying the Ring files to every Swift nodes. Once updated, the Ring
files are loaded by Swift processes without the need of being
restarted. Rollbacking data location is easily performed by
replacing the new Ring files by previous ones.</li>
</ul>
<p>However, being able to control the amount of data to move to a new
datacenter at a given step is a brand new feature, that has been fixed
in <a href="https://github.com/openstack/swift/releases/tag/2.2.0">version 2.2.0 of Swift</a>, released on October 4th of 2014.</p>
<h2>Checking data integrity</h2>
<p>Swift <code>auditor</code> processes (<code>swift-object-auditor</code>,
<code>swift-container-auditor</code> and <code>swift-account-auditor</code>) running on
storage nodes are continuously checking data integrity, by checking
files' checksums. When a corrupted file is found, it is <em>quarantined</em>;
the data is removed from the node and the <em>replication</em> mechanism
takes care of replacing the missing data. Below is an example of what
concretely happens when manually corrupting an object.</p>
<p>Let's corrupt data by hand:</p>
<div class="highlight"><pre><span></span><code>root@swnode0:/srv/node/d1/objects/154808/c3a#<span class="w"> </span>cat<span class="w"> </span>972e359caf9df6fdd3b8e295afd4cc3a/1410353767.57579.data
blabla
root@swnode0:/srv/node/d1/objects/154808/c3a#<span class="w"> </span><span class="nb">echo</span><span class="w"> </span>blablb<span class="w"> </span>><span class="w"> </span>972e359caf9df6fdd3b8e295afd4cc3a/1410353767.57579.data
</code></pre></div>
<p>The corrupted object is 'quarantined' by the object-auditor when it
checks the files integrity. Here's how it appears in the
<code>/var/log/syslog</code> log file:</p>
<div class="highlight"><pre><span></span><code>Sep 10 13:56:44 swnode0 object-auditor: Quarantined object /srv/node/d1/objects/154808/c3a/972e359caf9df6fdd3b8e295afd4cc3a/1410353767.57579.data: ETag 9b36b2e89df94bc458d629499d38cf86 and file's md5 6235440677e53f66877f0c1fec6a89bd do not match
Sep 10 13:56:44 swnode0 object-auditor: ERROR Object /srv/node/d1/objects/154808/c3a/972e359caf9df6fdd3b8e295afd4cc3a failed audit and was quarantined: ETag 9b36b2e89df94bc458d629499d38cf86 and file's md5 6235440677e53f66877f0c1fec6a89bd do not match
Sep 10 13:56:44 swnode0 object-auditor: Object audit (ALL) "forever" mode completed: 0.02s. Total quarantined: 1, Total errors: 0, Total files/sec: 46.71, Total bytes/sec: 326.94, Auditing time: 0.02, Rate: 0.98
</code></pre></div>
<p>The quarantined object is then overwritten by the object-replicator of
a node that has the appropriate replica uncorrupted. Below is an
extract of the log file on such node:</p>
<div class="highlight"><pre><span></span><code>Sep 10 13:57:01 swnode1 object-replicator: Starting object replication pass.
Sep 10 13:57:01 swnode1 object-replicator: <f+++++++++ c3a/972e359caf9df6fdd3b8e295afd4cc3a/1410353767.57579.data
Sep 10 13:57:01 swnode1 object-replicator: Successful rsync of /srv/node/d1/objects/154808/c3a at 192.168.100.10::object/d1/objects/154808 (0.182)
Sep 10 13:57:01 swnode1 object-replicator: 1/1 (100.00%) partitions replicated in 0.21s (4.84/sec, 0s remaining)
Sep 10 13:57:01 swnode1 object-replicator: 1 suffixes checked - 0.00% hashed, 100.00% synced
Sep 10 13:57:01 swnode1 object-replicator: Partition times: max 0.2050s, min 0.2050s, med 0.2050s
Sep 10 13:57:01 swnode1 object-replicator: Object replication complete. (0.00 minutes)
</code></pre></div>
<p>The corrupted data has been replaced by the correct data on the
initial storage node (where the file had been corrupted):</p>
<div class="highlight"><pre><span></span><code>root@swnode0:/srv/node/d1/objects/154808/c3a# cat 972e359caf9df6fdd3b8e295afd4cc3a/1410353767.57579.data
blabla
</code></pre></div>
<h2>Checking data location</h2>
<h3>Preparation</h3>
<p>We can use the <code>swift-dispersion-report</code> tool provided with Swift to
monitor our data dispersion ratio (ratio of objects on the proper
device / number of objects). A dedicated Openstack account is required
that will be used by <code>swift-dispersion-populate</code> to create containers
and objects.</p>
<p>Then we have to configure appropriately the <code>swift-dispersion-report</code>
tool with the <code>/etc/swift/dispersion.conf</code> file:</p>
<div class="highlight"><pre><span></span><code>[dispersion]
auth_url = http://SWIFT_PROXY_URL/auth/v1.0
auth_user = DEDICATED_ACCOUNT_USERNAME
auth_key = DEDICATED_ACCOUNT_PASSWORD
</code></pre></div>
<p>Once properly set, we can initiate dispersion monitoring by populating
our new account with test data:</p>
<div class="highlight"><pre><span></span><code>cloud@swproxy:~$<span class="w"> </span>swift-dispersion-populate
Created<span class="w"> </span><span class="m">2621</span><span class="w"> </span>containers<span class="w"> </span><span class="k">for</span><span class="w"> </span>dispersion<span class="w"> </span>reporting,<span class="w"> </span>4m,<span class="w"> </span><span class="m">0</span><span class="w"> </span>retries
Created<span class="w"> </span><span class="m">2621</span><span class="w"> </span>objects<span class="w"> </span><span class="k">for</span><span class="w"> </span>dispersion<span class="w"> </span>reporting,<span class="w"> </span>2m,<span class="w"> </span><span class="m">0</span><span class="w"> </span>retries
</code></pre></div>
<p>Our objects should have been placed on appropriate devices. We can
check this:</p>
<div class="highlight"><pre><span></span><code>cloud@swproxy:~$<span class="w"> </span>swift-dispersion-report
Queried<span class="w"> </span><span class="m">2622</span><span class="w"> </span>containers<span class="w"> </span><span class="k">for</span><span class="w"> </span>dispersion<span class="w"> </span>reporting,<span class="w"> </span>2m,<span class="w"> </span><span class="m">31</span><span class="w"> </span>retries
<span class="m">100</span>.00%<span class="w"> </span>of<span class="w"> </span>container<span class="w"> </span>copies<span class="w"> </span>found<span class="w"> </span><span class="o">(</span><span class="m">7866</span><span class="w"> </span>of<span class="w"> </span><span class="m">7866</span><span class="o">)</span>
Sample<span class="w"> </span>represents<span class="w"> </span><span class="m">1</span>.00%<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>container<span class="w"> </span>partition<span class="w"> </span>space
Queried<span class="w"> </span><span class="m">2621</span><span class="w"> </span>objects<span class="w"> </span><span class="k">for</span><span class="w"> </span>dispersion<span class="w"> </span>reporting,<span class="w"> </span>45s,<span class="w"> </span><span class="m">1</span><span class="w"> </span>retries
There<span class="w"> </span>were<span class="w"> </span><span class="m">2621</span><span class="w"> </span>partitions<span class="w"> </span>missing<span class="w"> </span><span class="m">0</span><span class="w"> </span>copy.
<span class="m">100</span>.00%<span class="w"> </span>of<span class="w"> </span>object<span class="w"> </span>copies<span class="w"> </span>found<span class="w"> </span><span class="o">(</span><span class="m">7863</span><span class="w"> </span>of<span class="w"> </span><span class="m">7863</span><span class="o">)</span>
Sample<span class="w"> </span>represents<span class="w"> </span><span class="m">1</span>.00%<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>object<span class="w"> </span>partition<span class="w"> </span>space
</code></pre></div>
<h3>Monitoring data redistribution</h3>
<p>Once updated ring has been pushed to every nodes and proxy servers, we
can follow the data redistribution with the
<code>swift-dispersion-report</code>. The migration is terminated when the number
of objects copies reach 100%. Here's an example of results obtained on
a 6 nodes cluster.</p>
<div class="highlight"><pre><span></span><code>cloud@swproxy:~$<span class="w"> </span>swift-dispersion-report
Queried<span class="w"> </span><span class="m">2622</span><span class="w"> </span>containers<span class="w"> </span><span class="k">for</span><span class="w"> </span>dispersion<span class="w"> </span>reporting,<span class="w"> </span>3m,<span class="w"> </span><span class="m">29</span><span class="w"> </span>retries
<span class="m">100</span>.00%<span class="w"> </span>of<span class="w"> </span>container<span class="w"> </span>copies<span class="w"> </span>found<span class="w"> </span><span class="o">(</span><span class="m">7866</span><span class="w"> </span>of<span class="w"> </span><span class="m">7866</span><span class="o">)</span>
Sample<span class="w"> </span>represents<span class="w"> </span><span class="m">1</span>.00%<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>container<span class="w"> </span>partition<span class="w"> </span>space
Queried<span class="w"> </span><span class="m">2621</span><span class="w"> </span>objects<span class="w"> </span><span class="k">for</span><span class="w"> </span>dispersion<span class="w"> </span>reporting,<span class="w"> </span>33s,<span class="w"> </span><span class="m">0</span><span class="w"> </span>retries
There<span class="w"> </span>were<span class="w"> </span><span class="m">23</span><span class="w"> </span>partitions<span class="w"> </span>missing<span class="w"> </span><span class="m">0</span><span class="w"> </span>copy.
There<span class="w"> </span>were<span class="w"> </span><span class="m">2598</span><span class="w"> </span>partitions<span class="w"> </span>missing<span class="w"> </span><span class="m">1</span><span class="w"> </span>copy.
<span class="m">66</span>.96%<span class="w"> </span>of<span class="w"> </span>object<span class="w"> </span>copies<span class="w"> </span>found<span class="w"> </span><span class="o">(</span><span class="m">5265</span><span class="w"> </span>of<span class="w"> </span><span class="m">7863</span><span class="o">)</span>
Sample<span class="w"> </span>represents<span class="w"> </span><span class="m">1</span>.00%<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>object<span class="w"> </span>partition<span class="w"> </span>space
<span class="c1"># Then some minutes later</span>
cloud@swproxy:~$<span class="w"> </span>swift-dispersion-report
Queried<span class="w"> </span><span class="m">2622</span><span class="w"> </span>containers<span class="w"> </span><span class="k">for</span><span class="w"> </span>dispersion<span class="w"> </span>reporting,<span class="w"> </span>5m,<span class="w"> </span><span class="m">0</span><span class="w"> </span>retries
<span class="m">100</span>.00%<span class="w"> </span>of<span class="w"> </span>container<span class="w"> </span>copies<span class="w"> </span>found<span class="w"> </span><span class="o">(</span><span class="m">7866</span><span class="w"> </span>of<span class="w"> </span><span class="m">7866</span><span class="o">)</span>
Sample<span class="w"> </span>represents<span class="w"> </span><span class="m">1</span>.00%<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>container<span class="w"> </span>partition<span class="w"> </span>space
Queried<span class="w"> </span><span class="m">2621</span><span class="w"> </span>objects<span class="w"> </span><span class="k">for</span><span class="w"> </span>dispersion<span class="w"> </span>reporting,<span class="w"> </span>26s,<span class="w"> </span><span class="m">0</span><span class="w"> </span>retries
There<span class="w"> </span>were<span class="w"> </span><span class="m">91</span><span class="w"> </span>partitions<span class="w"> </span>missing<span class="w"> </span><span class="m">0</span><span class="w"> </span>copy.
There<span class="w"> </span>were<span class="w"> </span><span class="m">2530</span><span class="w"> </span>partitions<span class="w"> </span>missing<span class="w"> </span><span class="m">1</span><span class="w"> </span>copy.
<span class="m">67</span>.82%<span class="w"> </span>of<span class="w"> </span>object<span class="w"> </span>copies<span class="w"> </span>found<span class="w"> </span><span class="o">(</span><span class="m">5333</span><span class="w"> </span>of<span class="w"> </span><span class="m">7863</span><span class="o">)</span>
Sample<span class="w"> </span>represents<span class="w"> </span><span class="m">1</span>.00%<span class="w"> </span>of<span class="w"> </span>the<span class="w"> </span>object<span class="w"> </span>partition<span class="w"> </span>space
</code></pre></div>
<h2>Limiting the amount of data to move</h2>
<p>There has been a number of recent contributions to Swift that have
been done in order to allow the smooth addition of nodes to a new
region.</p>
<p>With versions of <code>swift-ring-builder</code> earlier than Swift 2.1, when
adding a node to a new region, 1 replica of every object was moved to
the new region in order to maximize the dispersion of objects across
different regions. Such algorithm had severe drawbacks. Let's consider
a one region Swift cluster with 100 storage nodes. Adding 1 node to a
second region had the effect of transferring 1/3 of the cluster's data
to the new node, which would not have the capacity to store the data
previously distributed over 33 nodes. So in order to add a new region
to our cluster, we had to add in 1 step enough nodes to store 1/3 of
our data. Let's consider we add 33 nodes to the new region. While
there is enough capacity on these nodes to receive 1 replica of every
objects, such operation would trigger the transfer of Petabytes of
data to the new nodes. With a 10 Gigabits/second link between the 2
datacenters, such transfer would take days if not weeks, during which
the cluster's network and destination nodes' disks would be saturated.</p>
<p>With <a href="https://review.openstack.org/#/c/115441/">commit 6d77c37</a> ("Let admins add a region without melting
their cluster"), that has been released with Swift 2.1, the number of
partitions assigned to nodes in a new region was determined by the
weights of the nodes' devices. This feature allowed a Swift cluster
operator the limit the amount of data transferred to a new
region. However, because of <a href="https://bugs.launchpad.net/swift/+bug/1367826">bug 1367826</a> ("swift-ringbuilder
rebalance moves 100% partitions when adding a new node to a new
region"), even when limiting the amount of data transferred to a new
region, a big amount of data is moved uselessly inside the initial
region. For instance, it could happen that after a <code>swift-ring-builder
rebalance</code> operation, 3% of partitions were assigned to the new
region, but 88% of partitions were reassigned to new nodes inside the
first region. The would lead to uselessly loading the cluster's
network and storage nodes.</p>
<p>Eventually, <a href="https://review.openstack.org/#/c/121422/">commit 20e9ad5</a> ("Limit partition movement when adding
a new tier") fixed <a href="https://bugs.launchpad.net/swift/+bug/1367826">bug 1367826</a>. This commit has been released
with Swift 2.2. It allows an operator to choose the amount of data
that flows between regions, when adding nodes to a new region, without
border effects. This feature enables the operator to perform a multi
steps cluster split, by first adding devices with very low weights to
a new region, then by progressively increasing the weights step by
step. This can be done until 1 replica of every objects has been
transferred to the new region. Since the number of partitions assigned
to the new region depends on the weights assigned to the new devices,
the operator has to compute the appropriate weights.</p>
<h3>Computing new region weight for a given ratio of partitions</h3>
<p>In order to assign a given ratio of partitions to a new region, a
Swift operator can compute the devices' weights by using the following
formula.</p>
<p>Given:</p>
<ul>
<li>w1 is the weight of a single device in region r1</li>
<li>r1 has n1 devices</li>
<li>W1 = n1 * w1 is the full weight of region r1</li>
<li>r2 has n2 devices</li>
<li>w2 is the weight of a single device in region r2</li>
<li>W2 = n2 * w2 is the full weight of region r2</li>
<li>r is the ratio of partitions we want in region r2</li>
</ul>
<p>We have:</p>
<ul>
<li>r = W2 / (W1+W2)</li>
<li><=> W2 = r*W1 / (1-r)</li>
<li><=> w2 = r<em>W1 / (1-r)</em>n2</li>
</ul>
<p>w2 is the weight to set to each device of region r2</p>
<h3>Computing new devices weight for a given number of partitions</h3>
<p>In some cases the operator may prefer to specify the number of
partitions (rather than its ratio) that he wishes to assign to the
devices of a new region.</p>
<p>Given:</p>
<ul>
<li>p1 the number of partitions in region r1</li>
<li>W1 the full weight of region r1</li>
<li>p2 the number of partitions in region r2</li>
<li>W2 the full weight of region r2</li>
</ul>
<p>We have the following equality:</p>
<ul>
<li>p1/W1 = p2/W2</li>
<li><=> W2 = W1*p2 / p1</li>
<li><=> w2 = W1<em>p2 / n2</em>p1</li>
</ul>
<p>w2 is the weight to set to each device of region r2</p>
<h3>Some scripts to compute weights automatically</h3>
<p>I made some <a href="https://github.com/FlorentFlament/swift-scripts">Swift scripts</a> available to facilitate adding nodes to
a new region. <code>swift-add-nodes.py</code> allows adding nodes to a new region
with a minimal weight so that only 1 partition will be assigned to
each device (The number and names of devices is set in a constant at
the beginning of the script and has to be updated). Then
<code>swift-assign-partitions.py</code> allows assigning a chosen ratio of
partitions to the new region.</p>
<h2>Example of deployment</h2>
<p>Here's an example of the steps that a Swift operator can follow in
order to split its one region cluster into 2 regions smoothly. A first
step may consist in adding some new nodes to the new region and
assigning 1 partition to each device. This would typically move
between hundreds of Megabytes to a few Gigabytes; thus allowing to
check that everything (network, hardware, ...) is working as
expected. We can use the <code>swift-add-nodes.py</code> script to easily add
nodes to our new region with a minimal weight so that only 1 partition
will be assigned to each device:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>python<span class="w"> </span>swift-add-nodes.py<span class="w"> </span>object.builder<span class="w"> </span>object.builder.s1<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">6000</span><span class="w"> </span><span class="m">127</span>.0.0.1<span class="w"> </span><span class="m">127</span>.0.0.2<span class="w"> </span><span class="m">127</span>.0.0.3
Adding<span class="w"> </span>device:<span class="w"> </span><span class="o">{</span><span class="s1">'weight'</span>:<span class="w"> </span><span class="m">5</span>.11,<span class="w"> </span><span class="s1">'zone'</span>:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span><span class="s1">'ip'</span>:<span class="w"> </span><span class="s1">'127.0.0.1'</span>,<span class="w"> </span><span class="s1">'region'</span>:<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="s1">'device'</span>:<span class="w"> </span><span class="s1">'sdb1'</span>,<span class="w"> </span><span class="s1">'port'</span>:<span class="w"> </span><span class="m">6000</span><span class="o">}</span>
Adding<span class="w"> </span>device:<span class="w"> </span><span class="o">{</span><span class="s1">'weight'</span>:<span class="w"> </span><span class="m">5</span>.11,<span class="w"> </span><span class="s1">'zone'</span>:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span><span class="s1">'ip'</span>:<span class="w"> </span><span class="s1">'127.0.0.1'</span>,<span class="w"> </span><span class="s1">'region'</span>:<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="s1">'device'</span>:<span class="w"> </span><span class="s1">'sdc1'</span>,<span class="w"> </span><span class="s1">'port'</span>:<span class="w"> </span><span class="m">6000</span><span class="o">}</span>
Adding<span class="w"> </span>device:<span class="w"> </span><span class="o">{</span><span class="s1">'weight'</span>:<span class="w"> </span><span class="m">5</span>.11,<span class="w"> </span><span class="s1">'zone'</span>:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span><span class="s1">'ip'</span>:<span class="w"> </span><span class="s1">'127.0.0.1'</span>,<span class="w"> </span><span class="s1">'region'</span>:<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="s1">'device'</span>:<span class="w"> </span><span class="s1">'sdd1'</span>,<span class="w"> </span><span class="s1">'port'</span>:<span class="w"> </span><span class="m">6000</span><span class="o">}</span>
Adding<span class="w"> </span>device:<span class="w"> </span><span class="o">{</span><span class="s1">'weight'</span>:<span class="w"> </span><span class="m">5</span>.11,<span class="w"> </span><span class="s1">'zone'</span>:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span><span class="s1">'ip'</span>:<span class="w"> </span><span class="s1">'127.0.0.1'</span>,<span class="w"> </span><span class="s1">'region'</span>:<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="s1">'device'</span>:<span class="w"> </span><span class="s1">'sde1'</span>,<span class="w"> </span><span class="s1">'port'</span>:<span class="w"> </span><span class="m">6000</span><span class="o">}</span>
Adding<span class="w"> </span>device:<span class="w"> </span><span class="o">{</span><span class="s1">'weight'</span>:<span class="w"> </span><span class="m">5</span>.11,<span class="w"> </span><span class="s1">'zone'</span>:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span><span class="s1">'ip'</span>:<span class="w"> </span><span class="s1">'127.0.0.2'</span>,<span class="w"> </span><span class="s1">'region'</span>:<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="s1">'device'</span>:<span class="w"> </span><span class="s1">'sdb1'</span>,<span class="w"> </span><span class="s1">'port'</span>:<span class="w"> </span><span class="m">6000</span><span class="o">}</span>
Adding<span class="w"> </span>device:<span class="w"> </span><span class="o">{</span><span class="s1">'weight'</span>:<span class="w"> </span><span class="m">5</span>.11,<span class="w"> </span><span class="s1">'zone'</span>:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span><span class="s1">'ip'</span>:<span class="w"> </span><span class="s1">'127.0.0.2'</span>,<span class="w"> </span><span class="s1">'region'</span>:<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="s1">'device'</span>:<span class="w"> </span><span class="s1">'sdc1'</span>,<span class="w"> </span><span class="s1">'port'</span>:<span class="w"> </span><span class="m">6000</span><span class="o">}</span>
Adding<span class="w"> </span>device:<span class="w"> </span><span class="o">{</span><span class="s1">'weight'</span>:<span class="w"> </span><span class="m">5</span>.11,<span class="w"> </span><span class="s1">'zone'</span>:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span><span class="s1">'ip'</span>:<span class="w"> </span><span class="s1">'127.0.0.2'</span>,<span class="w"> </span><span class="s1">'region'</span>:<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="s1">'device'</span>:<span class="w"> </span><span class="s1">'sdd1'</span>,<span class="w"> </span><span class="s1">'port'</span>:<span class="w"> </span><span class="m">6000</span><span class="o">}</span>
Adding<span class="w"> </span>device:<span class="w"> </span><span class="o">{</span><span class="s1">'weight'</span>:<span class="w"> </span><span class="m">5</span>.11,<span class="w"> </span><span class="s1">'zone'</span>:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span><span class="s1">'ip'</span>:<span class="w"> </span><span class="s1">'127.0.0.2'</span>,<span class="w"> </span><span class="s1">'region'</span>:<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="s1">'device'</span>:<span class="w"> </span><span class="s1">'sde1'</span>,<span class="w"> </span><span class="s1">'port'</span>:<span class="w"> </span><span class="m">6000</span><span class="o">}</span>
Adding<span class="w"> </span>device:<span class="w"> </span><span class="o">{</span><span class="s1">'weight'</span>:<span class="w"> </span><span class="m">5</span>.11,<span class="w"> </span><span class="s1">'zone'</span>:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span><span class="s1">'ip'</span>:<span class="w"> </span><span class="s1">'127.0.0.3'</span>,<span class="w"> </span><span class="s1">'region'</span>:<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="s1">'device'</span>:<span class="w"> </span><span class="s1">'sdb1'</span>,<span class="w"> </span><span class="s1">'port'</span>:<span class="w"> </span><span class="m">6000</span><span class="o">}</span>
Adding<span class="w"> </span>device:<span class="w"> </span><span class="o">{</span><span class="s1">'weight'</span>:<span class="w"> </span><span class="m">5</span>.11,<span class="w"> </span><span class="s1">'zone'</span>:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span><span class="s1">'ip'</span>:<span class="w"> </span><span class="s1">'127.0.0.3'</span>,<span class="w"> </span><span class="s1">'region'</span>:<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="s1">'device'</span>:<span class="w"> </span><span class="s1">'sdc1'</span>,<span class="w"> </span><span class="s1">'port'</span>:<span class="w"> </span><span class="m">6000</span><span class="o">}</span>
Adding<span class="w"> </span>device:<span class="w"> </span><span class="o">{</span><span class="s1">'weight'</span>:<span class="w"> </span><span class="m">5</span>.11,<span class="w"> </span><span class="s1">'zone'</span>:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span><span class="s1">'ip'</span>:<span class="w"> </span><span class="s1">'127.0.0.3'</span>,<span class="w"> </span><span class="s1">'region'</span>:<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="s1">'device'</span>:<span class="w"> </span><span class="s1">'sdd1'</span>,<span class="w"> </span><span class="s1">'port'</span>:<span class="w"> </span><span class="m">6000</span><span class="o">}</span>
Adding<span class="w"> </span>device:<span class="w"> </span><span class="o">{</span><span class="s1">'weight'</span>:<span class="w"> </span><span class="m">5</span>.11,<span class="w"> </span><span class="s1">'zone'</span>:<span class="w"> </span><span class="m">0</span>,<span class="w"> </span><span class="s1">'ip'</span>:<span class="w"> </span><span class="s1">'127.0.0.3'</span>,<span class="w"> </span><span class="s1">'region'</span>:<span class="w"> </span><span class="m">2</span>,<span class="w"> </span><span class="s1">'device'</span>:<span class="w"> </span><span class="s1">'sde1'</span>,<span class="w"> </span><span class="s1">'port'</span>:<span class="w"> </span><span class="m">6000</span><span class="o">}</span>
$<span class="w"> </span>swift-ring-builder<span class="w"> </span>object.builder.s1<span class="w"> </span>rebalance
Reassigned<span class="w"> </span><span class="m">12</span><span class="w"> </span><span class="o">(</span><span class="m">0</span>.00%<span class="o">)</span><span class="w"> </span>partitions.<span class="w"> </span>Balance<span class="w"> </span>is<span class="w"> </span>now<span class="w"> </span><span class="m">0</span>.18.
</code></pre></div>
<p>Subsequent steps may consist in increasing the partitions count by
steps of some percentage (let's say 3%) until one third of total
cluster data is stored in the new region. Script
<code>swift-assign-partitions.py</code> allows assigning a chosen ratio of
partitions to the new region:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>python<span class="w"> </span>swift-assign-partitions.py<span class="w"> </span>object.builder.s2<span class="w"> </span>object.builder.s3<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">0</span>.03
Setting<span class="w"> </span>new<span class="w"> </span>weight<span class="w"> </span>of<span class="w"> </span><span class="m">10376</span>.28<span class="w"> </span>to<span class="w"> </span>device<span class="w"> </span><span class="m">1342</span>
Setting<span class="w"> </span>new<span class="w"> </span>weight<span class="w"> </span>of<span class="w"> </span><span class="m">10376</span>.28<span class="w"> </span>to<span class="w"> </span>device<span class="w"> </span><span class="m">1343</span>
Setting<span class="w"> </span>new<span class="w"> </span>weight<span class="w"> </span>of<span class="w"> </span><span class="m">10376</span>.28<span class="w"> </span>to<span class="w"> </span>device<span class="w"> </span><span class="m">1344</span>
Setting<span class="w"> </span>new<span class="w"> </span>weight<span class="w"> </span>of<span class="w"> </span><span class="m">10376</span>.28<span class="w"> </span>to<span class="w"> </span>device<span class="w"> </span><span class="m">1345</span>
Setting<span class="w"> </span>new<span class="w"> </span>weight<span class="w"> </span>of<span class="w"> </span><span class="m">10376</span>.28<span class="w"> </span>to<span class="w"> </span>device<span class="w"> </span><span class="m">1346</span>
Setting<span class="w"> </span>new<span class="w"> </span>weight<span class="w"> </span>of<span class="w"> </span><span class="m">10376</span>.28<span class="w"> </span>to<span class="w"> </span>device<span class="w"> </span><span class="m">1347</span>
Setting<span class="w"> </span>new<span class="w"> </span>weight<span class="w"> </span>of<span class="w"> </span><span class="m">10376</span>.28<span class="w"> </span>to<span class="w"> </span>device<span class="w"> </span><span class="m">1348</span>
Setting<span class="w"> </span>new<span class="w"> </span>weight<span class="w"> </span>of<span class="w"> </span><span class="m">10376</span>.28<span class="w"> </span>to<span class="w"> </span>device<span class="w"> </span><span class="m">1349</span>
Setting<span class="w"> </span>new<span class="w"> </span>weight<span class="w"> </span>of<span class="w"> </span><span class="m">10376</span>.28<span class="w"> </span>to<span class="w"> </span>device<span class="w"> </span><span class="m">1350</span>
Setting<span class="w"> </span>new<span class="w"> </span>weight<span class="w"> </span>of<span class="w"> </span><span class="m">10376</span>.28<span class="w"> </span>to<span class="w"> </span>device<span class="w"> </span><span class="m">1351</span>
Setting<span class="w"> </span>new<span class="w"> </span>weight<span class="w"> </span>of<span class="w"> </span><span class="m">10376</span>.28<span class="w"> </span>to<span class="w"> </span>device<span class="w"> </span><span class="m">1352</span>
Setting<span class="w"> </span>new<span class="w"> </span>weight<span class="w"> </span>of<span class="w"> </span><span class="m">10376</span>.28<span class="w"> </span>to<span class="w"> </span>device<span class="w"> </span><span class="m">1353</span>
$<span class="w"> </span>swift-ring-builder<span class="w"> </span>object.builder.s3<span class="w"> </span>rebalance
Reassigned<span class="w"> </span><span class="m">25119</span><span class="w"> </span><span class="o">(</span><span class="m">9</span>.58%<span class="o">)</span><span class="w"> </span>partitions.<span class="w"> </span>Balance<span class="w"> </span>is<span class="w"> </span>now<span class="w"> </span><span class="m">0</span>.25.
</code></pre></div>
<h2>Thanks & related links</h2>
<p>Special thanks to Christian Schwede for the awesome work he did to
improve the <code>swift-ring-builder</code>.</p>
<p>Interested in more details about <a href="http://www.florentflament.com/blog/openstack-swift-ring-made-understandable.html">how Openstack Swift Ring is
working</a> ?</p>
<p>Want to know more about all of this ? Come to see our talk <a href="https://openstacksummitnovember2014paris.sched.org/event/15afea20487919ec1e55c7f385df3f68#.VEVpSoXzTFY">Using
OpenStack Swift for Extreme Data Durability</a> at the next OpenStack
Summit in Paris !</p>OpenStack Swift Ring made understandable2014-10-11T22:00:00+02:002014-10-11T22:00:00+02:00Florent Flamenttag:www.florentflament.com,2014-10-11:/blog/openstack-swift-ring-made-understandable.html<p>When people talk about OpenStack Swift, we often hear the word
Ring. This is because the Ring is a central piece in how Swift is
working. But what <em>is</em> this thing everyone's talking about ?</p>
<p>The Ring refers to 3 files that are shared among every Swift nodes
(storage and proxy …</p><p>When people talk about OpenStack Swift, we often hear the word
Ring. This is because the Ring is a central piece in how Swift is
working. But what <em>is</em> this thing everyone's talking about ?</p>
<p>The Ring refers to 3 files that are shared among every Swift nodes
(storage and proxy nodes):</p>
<ul>
<li>object.ring.gz</li>
<li>container.ring.gz</li>
<li>account.ring.gz</li>
</ul>
<p>There is actually one ring per type of data manipulated by Swift:
Objects, Containers and Accounts. These files determine on which
physical devices (hard disks) will be stored each object (and also
each container and account). The number of devices on which an object
is stored depends on the number of replicas (copies) specified for the
Swift cluster.</p>
<h2>How does it concretely work</h2>
<p>When receiving an object to store, Swift computes a (MD5) hash of the
object's full name (including its account's and container's name). A
part of this hash is kept and interpreted by Swift as the partition
number. The length of the hash segment kept depends on the number of
partitions that has been set in the Swift cluster; This number is
necessarily a power of 2. So that if we keep n bits of the hash, we
have 2^n partitions.</p>
<p>The object ring is a map that associates each partition to a specific
physical device. This mechanism is then repeated for every object's
replicas, and also for containers and accounts.</p>
<p>To be more specific, the object's ring has 3 components:</p>
<ul>
<li>What is referred in the code as the <code>_replica2part2dev</code> table (which
name is relatively explicit as we'll see later on)</li>
<li>The table of devices describing each device</li>
<li>The length of an object's hash to consider as the partition number</li>
</ul>
<p>The <code>_replica2part2dev</code> structure is a 2-dimensional table, so that
for any (replica number, partition number) couple, the table indicates
the physical device, where the object should be stored.</p>
<p>The devices table contains every information that a Swift node needs
in order to reach a given device; It consists mostly in the device's
storage node's IP address, the TCP port to use, and the physical
device name on the storage node.</p>
<p>In the end, the Ring is composed of 2 tables and one integer. If I
were to choose a name for such structure, I would call it the Table. I
couldn't find any explanation of why the name Ring was adopted, but my
guess is that some previous algorithm may have used some modular
computation, which people tend to represent using rings..</p>
<h2>Example</h2>
<p>Here is a simple example to make everything clear. Let's consider a
Swift cluster with 2 storage nodes, with the following IPs addresses:
<code>192.168.0.10</code> and <code>192.168.0.11</code>. Each storage node has two devices:
<code>sdb1</code> and <code>sdc1</code>.</p>
<p>An example of <code>_replica2part2dev</code> table with 3 replicas, 8 partitions
and 4 devices would be:</p>
<div class="highlight"><pre><span></span><code>r
e | +-----------------+
p | 0 | 0 1 2 3 0 1 2 3 |
l | 1 | 1 2 3 0 1 2 3 0 |
i | 2 | 2 3 0 1 2 3 0 1 |
c v +-----------------+
a 0 1 2 3 4 5 6 7
------------------>
partition
</code></pre></div>
<p>The table has 3 lines, one for each replica, and 8 columns, one for
each partition. To find the device storing the replica number 1 of
partition number 2, we select the line of index 1 and column of index
2. This lead us to the device ID 3.</p>
<p>The devices table is very similar to what we can obtain by using the
<code>swift-ring-builder</code> with only the builder file as argument:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>swift-ring-builder<span class="w"> </span>mybuilder<span class="w"> </span>
mybuilder,<span class="w"> </span>build<span class="w"> </span>version<span class="w"> </span><span class="m">4</span>
<span class="m">8</span><span class="w"> </span>partitions,<span class="w"> </span><span class="m">3</span>.000000<span class="w"> </span>replicas,<span class="w"> </span><span class="m">1</span><span class="w"> </span>regions,<span class="w"> </span><span class="m">1</span><span class="w"> </span>zones,<span class="w"> </span><span class="m">4</span><span class="w"> </span>devices,<span class="w"> </span><span class="m">0</span>.00<span class="w"> </span>balance
The<span class="w"> </span>minimum<span class="w"> </span>number<span class="w"> </span>of<span class="w"> </span>hours<span class="w"> </span>before<span class="w"> </span>a<span class="w"> </span>partition<span class="w"> </span>can<span class="w"> </span>be<span class="w"> </span>reassigned<span class="w"> </span>is<span class="w"> </span><span class="m">0</span>
Devices:<span class="w"> </span>id<span class="w"> </span>region<span class="w"> </span>zone<span class="w"> </span>ip<span class="w"> </span>address<span class="w"> </span>port<span class="w"> </span>replication<span class="w"> </span>ip<span class="w"> </span>replication<span class="w"> </span>port<span class="w"> </span>name<span class="w"> </span>weight<span class="w"> </span>partitions<span class="w"> </span>balance<span class="w"> </span>meta
<span class="w"> </span><span class="m">0</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">192</span>.168.0.10<span class="w"> </span><span class="m">6000</span><span class="w"> </span><span class="m">192</span>.168.0.10<span class="w"> </span><span class="m">6000</span><span class="w"> </span>sdb1<span class="w"> </span><span class="m">100</span>.00<span class="w"> </span><span class="m">6</span><span class="w"> </span><span class="m">0</span>.00<span class="w"> </span>
<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">192</span>.168.0.10<span class="w"> </span><span class="m">6000</span><span class="w"> </span><span class="m">192</span>.168.0.10<span class="w"> </span><span class="m">6000</span><span class="w"> </span>sdc1<span class="w"> </span><span class="m">100</span>.00<span class="w"> </span><span class="m">6</span><span class="w"> </span><span class="m">0</span>.00<span class="w"> </span>
<span class="w"> </span><span class="m">2</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">192</span>.168.0.11<span class="w"> </span><span class="m">6000</span><span class="w"> </span><span class="m">192</span>.168.0.11<span class="w"> </span><span class="m">6000</span><span class="w"> </span>sdb1<span class="w"> </span><span class="m">100</span>.00<span class="w"> </span><span class="m">6</span><span class="w"> </span><span class="m">0</span>.00<span class="w"> </span>
<span class="w"> </span><span class="m">3</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="m">192</span>.168.0.11<span class="w"> </span><span class="m">6000</span><span class="w"> </span><span class="m">192</span>.168.0.11<span class="w"> </span><span class="m">6000</span><span class="w"> </span>sdc1<span class="w"> </span><span class="m">100</span>.00<span class="w"> </span><span class="m">6</span><span class="w"> </span><span class="m">0</span>.00
</code></pre></div>
<p>The device of ID 3 can be found on server 192.168.0.11, port 6000,
device name sdc1.</p>
<h2>Simple is good</h2>
<p>What I like with such mechanism is that the smartness of the data
placement is performed by the <code>swift-ring-builder</code>, a standalone tool
provided with Swift. Once the rings have been built, Swift processes
running on the Swift nodes have a fully deterministic and easily
predictable behavior.</p>
<p>The <code>swift-ring-builder</code> manipulates <code>builders</code> files; these are files
containing architectural information about the Swift cluster (like
distribution of devices and nodes among regions and zones). These
<code>builders</code> are then used to generate the <code>rings</code> files. As with the
rings, there is one builder per type of data (objects, containers and
accounts).</p>
<p>Thanks to this mechanism the complexity of smartly storing objects has
been well separated between:</p>
<ul>
<li>
<p>Smartly assigning partitions (and corresponding objects, containers
and accounts) to devices, taking into account the cluster's
architecture. This is performed by the <code>swift-ring-builder</code></p>
</li>
<li>
<p>Ensuring that files are stored uncorrupted at the appropriate
locations; This is performer by the processes running on the Swift
nodes.</p>
</li>
</ul>
<h2>More</h2>
<p>For more information about the ring, one can read the <a href="http://docs.openstack.org/developer/swift/overview_ring.html">Swift's
developer documentation about the Ring</a>.</p>Windows Images for OpenStack2014-03-30T19:15:00+02:002014-03-30T19:15:00+02:00Florent Flamenttag:www.florentflament.com,2014-03-30:/blog/windows-images-for-openstack.html<p>This note summarizes articles from other places about Microsoft
Windows images for OpenStack creation, along with some first hand
experience. The whole process of creating Windows 2008 and Windows
2012 images fully usable on OpenStack instances is described there.</p>
<h1>Prerequisite</h1>
<p>To achieve the creation of a qcow2 Windows image for …</p><p>This note summarizes articles from other places about Microsoft
Windows images for OpenStack creation, along with some first hand
experience. The whole process of creating Windows 2008 and Windows
2012 images fully usable on OpenStack instances is described there.</p>
<h1>Prerequisite</h1>
<p>To achieve the creation of a qcow2 Windows image for OpenStack, we
need the following ISO images:</p>
<ul>
<li>
<p>An ISO image of the installation DVD for all OSes that we wish to
use in OpenStack. These ISOs can usually be downloaded from a
company's account on Microsoft website, once the appropriate
contract has been signed. For testing purpose, <a href="http://technet.microsoft.com/en-us/evalcenter/hh670538.aspx">Windows Server 2012
Evaluation ISO</a> can be downloaded on Microsoft website.</p>
</li>
<li>
<p>The latest <a href="http://www.linux-kvm.org/page/WindowsGuestDrivers/Download_Drivers">VirtIO drivers for Windows</a>. These are optimized
drivers to run Windows OSes with KVM virtualized hard disk
controller and network devices.</p>
</li>
</ul>
<h1>Base image creation</h1>
<p>The first step to build a Microsoft Windows image is to install the OS
in a VM as we would have done on a bare metal computer. <a href="http://blog.gridcentric.com/bid/297627/Creating-a-Windows-Image-on-OpenStack">Gridcentric's
article about Windows image creation</a> describes this procedure in
details.</p>
<p>The steps to follow are:</p>
<ul>
<li>
<p>Create an empty qcow2 image (this will be the disk on which we'll
install our OS). I typically use a 9 GB image for Windows 2008, and
a 17 GB for Windows 2012 (although I think it should work with a 11
or 12 GB image). Example:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>qemu-img<span class="w"> </span>create<span class="w"> </span>-f<span class="w"> </span>qcow2<span class="w"> </span>Windows-Server-2008-R2.qcow2<span class="w"> </span>9G
</code></pre></div>
</li>
<li>
<p>Next step is to launch Windows' installation in a (KVM) virtual
machine. The following command is an example for that:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kvm<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-m<span class="w"> </span><span class="m">2048</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-cdrom<span class="w"> </span><WINDOWS_INSTALLER_ISO><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-drive<span class="w"> </span><span class="nv">file</span><span class="o">=</span>Windows-Server-2008-R2.qcow2,if<span class="o">=</span>virtio<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-drive<span class="w"> </span><span class="nv">file</span><span class="o">=</span><VIRTIO_DRIVERS_ISO>,index<span class="o">=</span><span class="m">3</span>,media<span class="o">=</span>cdrom<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-net<span class="w"> </span>nic,model<span class="o">=</span>virtio<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-net<span class="w"> </span>user<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-nographic<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-vnc<span class="w"> </span>:9<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-k<span class="w"> </span>fr<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-usbdevice<span class="w"> </span>tablet
</code></pre></div>
</li>
<li>
<p>The following step consists in connecting to the display of the VM
launched previously through VNC, in order to manually pursue the
installation. This can be done with the following command:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>xvncviewer<span class="w"> </span><IP_OF_HYPERVISOR>:9
</code></pre></div>
</li>
<li>
<p>During the installation, Windows will ask for the Hard Disk
controller driver. We have to select the VirtIO driver, which is
located on the VirtIO CDROM (WIN7 directory for Windows 2008,
and WIN8 directory for Windows 2012).</p>
</li>
<li>
<p>Once the basic Windows installation is done, we have to set the
appropriate Network device driver in the Windows Devices
Manager. The Network device VirtIO driver is available in the same
directory than the Hard Disk controller driver specified in the
previous step.</p>
</li>
<li>
<p>Since VMs will be managed by RDP, we have to activate the
service. This is done by navigating through the following menu:
Computer (right-lick) -> Properties -> remote tab, and selecting the
following option: allow connections from computers running any
version of Remote Desktop.</p>
</li>
<li>
<p>An additional step consisting in opening the appropriate Firewall
ports is required on Windows 2012: Network (right-click) ->
Properties -> Windows Firewall -> Advanced Settings -> Inbound
rules. Then enable: Remote Desktop - Shadow, Remote Desktop - User
Mode TCP, Remote Desktop - User Mode UDP.</p>
</li>
</ul>
<h1>Customizing image for OpenStack</h1>
<p>The previous steps allowed us to have Windows fully installed in a KVM
virtual machine. The last steps consist in installing <a href="http://www.cloudbase.it/cloud-init-for-windows-instances/">Cloud-Init for
Windows</a>, a Windows implementation of the Linux based
<a href="http://cloudinit.readthedocs.org/en/latest/">Cloud-Init</a> mechanism. This set of scripts transforms a legacy OS
image into a ready for OpenStack image. At instantiation of a VM,
Cloud-Init fetches from a meta-data server, data such as ssh public
key and hostname that allows the instance to become unique. Cloud-Init
base is Open source, and Cloudbase provides an <a href="https://www.cloudbase.it/downloads/CloudbaseInitSetup_Beta.msi">installer</a> on its
blog. We'll install Cloud-Init by injecting the installer that we just
downloaded. To to that, we have to follow these steps:</p>
<ul>
<li>
<p>Shutdown Windows</p>
</li>
<li>
<p>Mount the qcow2 image on the hypervisor filesystem, then copy the
installer on Windows' administrator desktop, with something like:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>sudo<span class="w"> </span>qemu-nbd<span class="w"> </span>-c<span class="w"> </span>/dev/nbd2<span class="w"> </span>Windows-Server-2008-R2.qcow2
$<span class="w"> </span>sudo<span class="w"> </span>mount<span class="w"> </span>/dev/nbd2p2<span class="w"> </span>mnt/
$<span class="w"> </span>cp<span class="w"> </span><INSTALLER><span class="w"> </span><ADMINISTRATOR_DESKTOP_ON_WINDOWS>
$<span class="w"> </span>sudo<span class="w"> </span>umount<span class="w"> </span>mnt/
$<span class="w"> </span>sudo<span class="w"> </span>qemu-nbd<span class="w"> </span>-d<span class="w"> </span>/dev/nbd2
</code></pre></div>
</li>
<li>
<p>Restart Windows in KVM with the same command that we used to install
Windows in the first place:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>kvm<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-m<span class="w"> </span><span class="m">2048</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-cdrom<span class="w"> </span><WINDOWS_INSTALLER_ISO><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-drive<span class="w"> </span><span class="nv">file</span><span class="o">=</span>Windows-Server-2008-R2.qcow2,if<span class="o">=</span>virtio<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-drive<span class="w"> </span><span class="nv">file</span><span class="o">=</span><VIRTIO_DRIVERS_ISO>,index<span class="o">=</span><span class="m">3</span>,media<span class="o">=</span>cdrom<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-net<span class="w"> </span>nic,model<span class="o">=</span>virtio<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-net<span class="w"> </span>user<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-nographic<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-vnc<span class="w"> </span>:9<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-k<span class="w"> </span>fr<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-usbdevice<span class="w"> </span>tablet
</code></pre></div>
</li>
<li>
<p>Then connect again with xvncviewer:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>xvncviewer<span class="w"> </span><IP_OF_HYPERVISOR>:9
</code></pre></div>
</li>
<li>
<p>This time, we have to launch the CloudbaseInitSetup_Beta.msi
installer, and follow the instructions as described on <a href="http://www.cloudbase.it/cloud-init-for-windows-instances/">Cloudbase
blog</a>. At the end of the installation, we have to check the "run
sysprep" option, but not the "shutdown" one. Sysprep is the tool
provided by Microsoft to make a VM unique (generates a unique OS ID
to be used for some Microsoft services), once it's instantiated.</p>
</li>
<li>
<p>Once the installation is done, we can clean any temporary files
created, then shutdown the system. The image is ready to be uploaded
in OpenStack Glance:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>glance<span class="w"> </span>add<span class="w"> </span><span class="nv">name</span><span class="o">=</span><OPENSTACK_IMAGE_NAME><span class="w"> </span><span class="nv">is_public</span><span class="o">=</span><span class="nb">true</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="nv">container_format</span><span class="o">=</span>bare<span class="w"> </span><span class="nv">disk_format</span><span class="o">=</span>qcow2<span class="w"> </span><<span class="w"> </span><IMAGE_FILENAME>
</code></pre></div>
</li>
</ul>
<h1>Connecting to a Windows VM</h1>
<p>The usual mechanism used in OpenStack to connect to VMs (running
Linux) is ssh. A public key specified by the user launching the VM is
set in the default user's <code>authorized_keys</code> file. This allows the user
to connect to the VM by using the corresponding private key.</p>
<p>However, it is not currently possible to connect to a Windows VM with
ssh (there is some <a href="http://www.cloudbase.it/windows-without-passwords-in-openstack/">work done in this direction</a> that I've not
tested yet). Cloud-Init base creates an <code>Admin</code> user, with either:</p>
<ul>
<li>
<p>a password specified by the user on the command line. Note that the
password must respect Windows password strength constraints (upper
and lower case characters, as well as numbers). If not it will be
silently ignored. For instance:</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>nova<span class="w"> </span>boot<span class="w"> </span>--key-name<span class="w"> </span><KEYPAIR_NAME><span class="w"> </span>--image<span class="w"> </span><IMAGE_ID><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--flavor<span class="w"> </span><FLAVOR_ID><span class="w"> </span>--nic<span class="w"> </span>net-id<span class="o">=</span><NET_ID><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>--meta<span class="w"> </span><span class="nv">admin_pass</span><span class="o">=</span><ADMIN_PASSWORD><span class="w"> </span><VM_NAME>
</code></pre></div>
</li>
<li>
<p>a password automatically generated during VM instantiation, and
encrypted with the ssh public key provided when launching the
VM. Such password can be retrieved and decrypted with the
corresponding private ssh key, by using the following command (note
that the private key is used locally):</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>nova<span class="w"> </span>get-password<span class="w"> </span><VM_NAME_OR_ID><span class="w"> </span><SSH_PRIVATE_KEY>
</code></pre></div>
</li>
</ul>
<p>Note that to connect to a Windows VM from Linux, prefer using
<code>xfreerdp</code> instead of <code>rdesktop</code>. The pointer is bogus when connecting
to a windows 2012 VM using <code>rdesktop</code>.</p>Customizing OpenStack RBAC policies2014-02-09T00:00:00+01:002014-02-09T00:00:00+01:00Florent Flamenttag:www.florentflament.com,2014-02-09:/blog/customizing-openstack-rbac-policies.html<p>OpenStack uses a <a href="http://docs.openstack.org/developer/keystone/configuration.html#keystone-api-protection-with-role-based-access-control-rbac">role based access control</a> (RBAC) mechanism to
manage accesses to its resources. With the current architecture,
users' roles granted on each project and domain are stored into
Keystone, and can be updated through <a href="http://api.openstack.org/api-ref-identity.html#identity-v3">Keystone's API</a>. However,
policy enforcement (actually allowing or not the access to resources
according …</p><p>OpenStack uses a <a href="http://docs.openstack.org/developer/keystone/configuration.html#keystone-api-protection-with-role-based-access-control-rbac">role based access control</a> (RBAC) mechanism to
manage accesses to its resources. With the current architecture,
users' roles granted on each project and domain are stored into
Keystone, and can be updated through <a href="http://api.openstack.org/api-ref-identity.html#identity-v3">Keystone's API</a>. However,
policy enforcement (actually allowing or not the access to resources
according to a user's roles) is performed independently in each
service, based on the rules defined in each <code>policy.json</code> file.</p>
<p>In a default OpenStack setup (like Devstack), two roles are created:</p>
<ul>
<li>
<p>The <code>Member</code> role, which when granted to a user on a project, allows
him to manage resources (instances, volumes, ...) in this project.</p>
</li>
<li>
<p>The <code>admin</code> role, which when granted to a user on any project,
offers to this user a total control over the whole OpenStack
platform. Although this is the current behavior, it has been <a href="https://bugs.launchpad.net/keystone/+bug/968696">marked
as a bug</a>.</p>
</li>
</ul>
<p>However, the OpenStack policy engine allows operators to specify fine
grained set of rules to control access to resources of each OpenStack
service (Keystone, Nova, Cinder, ...).</p>
<h2>Attributes available to build custom policies</h2>
<p>Four types of attributes can be used to set policy rules:</p>
<ul>
<li>
<p>User roles, which can be checked by using the following syntax:</p>
<div class="highlight"><pre><span></span><code>role:<requires_role>
</code></pre></div>
</li>
<li>
<p>Other user related attributes (stored into or obtained through the
token). The following attributes are available: user_id, domain_id
or project_id (depending on the scope), and can be checked against
constants or other attributes:</p>
<div class="highlight"><pre><span></span><code>project_id:<some_attribute>
</code></pre></div>
</li>
<li>
<p>API call attributes are any data sent along with the API call. They
can be checked against constants or user attributes. For instance,
the following statement checks that a user being created is in the
same domain as his creator (note that API call attributes have to be
on the right side of the expression, while user attributes are on
the left side):</p>
<div class="highlight"><pre><span></span><code>domain_id:user.domain_id
</code></pre></div>
</li>
<li>
<p>The fourth category of attributes are what I'd call contextual
attributes. These are the attributes of objects referenced (or
targeted) by an API call; i.e. any object whose id appear somewhere
in the API call. For instance, when granting a new role on a project
to a user, all attributes related to the role, the project and the
user are available to the policy engine, through the <code>target</code>
keyword. The following syntax checks that the role of the context is
the <code>Member</code> role:</p>
<div class="highlight"><pre><span></span><code>'Member':target.role.name
</code></pre></div>
</li>
</ul>
<p>Depending on the type of API calls, some of the following attributes
will be available, according to the objects impacted by the action:</p>
<ul>
<li>
<p>domain:</p>
<ul>
<li>target.domain.enabled</li>
<li>target.domain.id</li>
<li>target.domain.name</li>
</ul>
</li>
<li>
<p>group:</p>
<ul>
<li>target.group.description</li>
<li>target.group.domain_id</li>
<li>target.group.id</li>
<li>target.group.name</li>
</ul>
</li>
<li>
<p>project:</p>
<ul>
<li>target.project.description</li>
<li>target.project.domain_id</li>
<li>target.project.enabled</li>
<li>target.project.id</li>
<li>target.project.name</li>
</ul>
</li>
<li>
<p>role:</p>
<ul>
<li>target.role.id</li>
<li>target.role.name</li>
</ul>
</li>
<li>
<p>user:</p>
<ul>
<li>target.user.default_project_id</li>
<li>target.user.description</li>
<li>target.user.domain_id</li>
<li>target.user.enabled</li>
<li>target.user.id</li>
<li>target.user.name</li>
</ul>
</li>
</ul>
<h2>Example: admin and super_admin</h2>
<p>The following example is taken from a User Story that we were
considering at <a href="http://www.cloudwatt.com">CloudWatt</a>. As a cloud service provider, we wanted
to be able to have 2 different levels of administrator roles:</p>
<ul>
<li>An <code>admin</code> role, which allows its users to grant the <code>Member</code> role
to any other user.</li>
<li>While the <code>super_admin</code> role allows granting any role.</li>
</ul>
<p>When added to Keystone's ̀<code>policy.json</code> file, the following rules
implements the two roles described previously:</p>
<div class="highlight"><pre><span></span><code>"admin_grant_member": "role:admin and 'Member':%(target.role.name)s",
"identity:create_grant": "role:super_admin or rule:admin_grant_member",
</code></pre></div>
<p>The first rule describes a new rule called <code>admin_grant_member</code>, which
checks that the user authenticated by the token has the <code>admin</code> role
(on its scope), and that the role in the context (the role the admin
is trying to grant) is the <code>Member</code> role (we used the <code>name</code>
attribute, but could use the role's id instead).</p>
<p>The second rule is checked whenever an API call is made to grant a
role to a user (action <code>identity:create_grant</code>). This rule tells the
policy engine that in order for a user to be allowed to grant a role
to another user, the user authenticated by the token must either have
the <code>super_admin</code> role, or satisfy the <code>admin_grant_member</code> rule.</p>
<p>Put together these two rules actually meet the use case. Any user with
the <code>admin</code> role will only be able to grant the <code>Member</code> role to other
users, while users with the <code>super_admin</code> role will be able to grant
any role.</p>
<h2>Notes</h2>
<p>One of the most powerful rules that the OpenStack policy engine
allows, are those limiting a user's actions to his own domain or
project. These kind of rules are widely used in <a href="https://github.com/openstack/keystone/blob/master/etc/policy.v3cloudsample.json">Keystone's
policy.v3cloudsample.json</a>.</p>
<p>Also note, that a recent patch merged into oslo-incubator implements
the blueprint allowing the policy engine to <a href="https://blueprints.launchpad.net/oslo/+spec/policy-constant-check">check contextual
attributes against constant values</a>. This patch will have to be
synchronized into the OpenStack projects for them to benefit from this
feature.</p>Setting Keystone v3 domains2014-01-18T00:00:00+01:002014-01-18T00:00:00+01:00Florent Flamenttag:www.florentflament.com,2014-01-18:/blog/setting-keystone-v3-domains.html<p>The <a href="http://api.openstack.org/api-ref-identity.html#identity-v3">Openstack Identity v3 API</a>, provided by Keystone, offers
features that were lacking in the previous version. Among these
features, it introduces the concept of domains, allowing isolation of
projects and users. For instance, an administrator allowed to create
projects and users in a given domain, may not have any …</p><p>The <a href="http://api.openstack.org/api-ref-identity.html#identity-v3">Openstack Identity v3 API</a>, provided by Keystone, offers
features that were lacking in the previous version. Among these
features, it introduces the concept of domains, allowing isolation of
projects and users. For instance, an administrator allowed to create
projects and users in a given domain, may not have any right in
another one. While these features look very exciting, some
configuration needs to be done to have a working identity v3 service
with domains properly set.</p>
<p><a href="http://docs.openstack.org/developer/keystone/configuration.html#keystone-api-protection-with-role-based-access-control-rbac">Keystone API protection</a> section of the developer's doc provides
hints about how to set-up a multi-domain installation. Starting from
there, I describe the full steps to have a multi-domain setup running,
by using <code>curl</code> to send http requests and <code>jq</code> to parse the json
answers.</p>
<h2>Setting an admin domain and a cloud admin</h2>
<p>First, we have to start on a fresh non multi-domain installation with
the <a href="https://github.com/openstack/keystone/blob/master/etc/policy.json">default policy file</a>.</p>
<ul>
<li>
<p>With the <code>admin</code> user we can create the <code>admin_domain</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nv">ADMIN_TOKEN</span><span class="o">=</span><span class="k">$(</span><span class="se">\</span>
curl<span class="w"> </span>http://localhost:5000/v3/auth/tokens<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-i<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type: application/json"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">'</span>
<span class="s1">{</span>
<span class="s1"> "auth": {</span>
<span class="s1"> "identity": {</span>
<span class="s1"> "methods": [</span>
<span class="s1"> "password"</span>
<span class="s1"> ],</span>
<span class="s1"> "password": {</span>
<span class="s1"> "user": {</span>
<span class="s1"> "domain": {</span>
<span class="s1"> "name": "Default"</span>
<span class="s1"> },</span>
<span class="s1"> "name": "admin",</span>
<span class="s1"> "password": "password"</span>
<span class="s1"> }</span>
<span class="s1"> }</span>
<span class="s1"> },</span>
<span class="s1"> "scope": {</span>
<span class="s1"> "project": {</span>
<span class="s1"> "domain": {</span>
<span class="s1"> "name": "Default"</span>
<span class="s1"> },</span>
<span class="s1"> "name": "admin"</span>
<span class="s1"> }</span>
<span class="s1"> }</span>
<span class="s1"> }</span>
<span class="s1">}'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>^X-Subject-Token:<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{print $2}'</span><span class="w"> </span><span class="k">)</span>
<span class="nv">ID_ADMIN_DOMAIN</span><span class="o">=</span><span class="k">$(</span><span class="se">\</span>
curl<span class="w"> </span>http://localhost:5000/v3/domains<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$ADMIN_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type: application/json"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">'</span>
<span class="s1">{</span>
<span class="s1"> "domain": {</span>
<span class="s1"> "enabled": true,</span>
<span class="s1"> "name": "admin_domain"</span>
<span class="s1"> }</span>
<span class="s1">}'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.domain.id<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'"'</span><span class="w"> </span><span class="k">)</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"ID of domain cloud: </span><span class="nv">$ID_ADMIN_DOMAIN</span><span class="s2">"</span>
</code></pre></div>
</li>
<li>
<p>Then we can create our <code>cloud_admin</code> user, within the <code>admin_domain</code>
domain.</p>
<div class="highlight"><pre><span></span><code><span class="nv">ID_CLOUD_ADMIN</span><span class="o">=</span><span class="k">$(</span><span class="se">\</span>
curl<span class="w"> </span>http://localhost:5000/v3/users<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$ADMIN_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type: application/json"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span>
<span class="s2">{</span>
<span class="s2"> \"user\": {</span>
<span class="s2"> \"description\": \"Cloud administrator\",</span>
<span class="s2"> \"domain_id\": \"</span><span class="nv">$ID_ADMIN_DOMAIN</span><span class="s2">\",</span>
<span class="s2"> \"enabled\": true,</span>
<span class="s2"> \"name\": \"cloud_admin\",</span>
<span class="s2"> \"password\": \"password\"</span>
<span class="s2"> }</span>
<span class="s2">}"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.user.id<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'"'</span><span class="w"> </span><span class="k">)</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"ID of user cloud_admin: </span><span class="nv">$ID_CLOUD_ADMIN</span><span class="s2">"</span>
</code></pre></div>
</li>
<li>
<p>And we grant to our user <code>cloud_admin</code> the <code>admin</code> role on domain
<code>admin_domain</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nv">ADMIN_ROLE_ID</span><span class="o">=</span><span class="k">$(</span><span class="se">\</span>
curl<span class="w"> </span>http://localhost:5000/v3/roles?name<span class="o">=</span>admin<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$ADMIN_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.roles<span class="o">[</span><span class="m">0</span><span class="o">]</span>.id<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'"'</span><span class="w"> </span><span class="k">)</span>
curl<span class="w"> </span>-X<span class="w"> </span>PUT<span class="w"> </span>http://localhost:5000/v3/domains/<span class="si">${</span><span class="nv">ID_ADMIN_DOMAIN</span><span class="si">}</span>/users/<span class="si">${</span><span class="nv">ID_CLOUD_ADMIN</span><span class="si">}</span>/roles/<span class="si">${</span><span class="nv">ADMIN_ROLE_ID</span><span class="si">}</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-i<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$ADMIN_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type: application/json"</span>
curl<span class="w"> </span>http://localhost:5000/v3/domains/<span class="si">${</span><span class="nv">ID_ADMIN_DOMAIN</span><span class="si">}</span>/users/<span class="si">${</span><span class="nv">ID_CLOUD_ADMIN</span><span class="si">}</span>/roles<span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$ADMIN_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.roles
</code></pre></div>
</li>
<li>
<p>Once the <code>admin_domain</code> has been created with its <code>cloud_admin</code>
user, we can enforce a domain based policy. In order to do that, we
have to copy the <a href="https://github.com/openstack/keystone/blob/master/etc/policy.v3cloudsample.json">policy.v3cloudsample.json</a> file over our former
<code>/etc/keystone/policy.json</code>, while replacing the string
<code>admin_domain_id</code> by the ID of the <code>admin_domain</code> we just
created. Locate the <code>policy.v3cloudsample.json</code> file into the <code>etc</code>
directory of Keystone's source.</p>
<div class="highlight"><pre><span></span><code>sed<span class="w"> </span>s/admin_domain_id/<span class="si">${</span><span class="nv">ID_ADMIN_DOMAIN</span><span class="si">}</span>/<span class="w"> </span><span class="se">\</span>
<span class="w"> </span><<span class="w"> </span>policy.v3cloudsample.json<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>><span class="w"> </span>/etc/keystone/policy.json
</code></pre></div>
</li>
</ul>
<p>Warning, current version (commit
19620076f587f925c5d2fa59780c1a80dde15db2) of policy.v3cloudsample.json
doesn't allow <code>cloud_admin</code> to manage users in other domains than its
own (see <a href="https://bugs.launchpad.net/keystone/+bug/1267187">bug 1267187</a>). Until the patch is merged, I suggest using
this <a href="https://review.openstack.org/#/c/65510/">policy.c3cloudsample.json under review</a>.</p>
<h2>Creating domains and admins</h2>
<p>From now on, the <code>admin</code> user can only manage projects and users in
the <code>Default</code> domain. To create other domains we will have to
authenticate with the <code>cloud_admin</code> user created above.</p>
<ul>
<li>
<p>Getting a token scoped on the <code>admin_domain</code>, for user <code>cloud_admin</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nv">CLOUD_ADMIN_TOKEN</span><span class="o">=</span><span class="k">$(</span><span class="se">\</span>
curl<span class="w"> </span>http://localhost:5000/v3/auth/tokens<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-i<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type: application/json"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">'</span>
<span class="s1">{</span>
<span class="s1"> "auth": {</span>
<span class="s1"> "identity": {</span>
<span class="s1"> "methods": [</span>
<span class="s1"> "password"</span>
<span class="s1"> ],</span>
<span class="s1"> "password": {</span>
<span class="s1"> "user": {</span>
<span class="s1"> "domain": {</span>
<span class="s1"> "name": "admin_domain"</span>
<span class="s1"> },</span>
<span class="s1"> "name": "cloud_admin",</span>
<span class="s1"> "password": "password"</span>
<span class="s1"> }</span>
<span class="s1"> }</span>
<span class="s1"> },</span>
<span class="s1"> "scope": {</span>
<span class="s1"> "domain": {</span>
<span class="s1"> "name": "admin_domain"</span>
<span class="s1"> }</span>
<span class="s1"> }</span>
<span class="s1"> }</span>
<span class="s1">}'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>^X-Subject-Token:<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{print $2}'</span><span class="w"> </span><span class="k">)</span>
</code></pre></div>
</li>
<li>
<p>Creating domains <code>dom1</code> and <code>dom2</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nv">ID_DOM1</span><span class="o">=</span><span class="k">$(</span><span class="se">\</span>
curl<span class="w"> </span>http://localhost:5000/v3/domains<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$CLOUD_ADMIN_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type: application/json"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">'</span>
<span class="s1">{</span>
<span class="s1"> "domain": {</span>
<span class="s1"> "enabled": true,</span>
<span class="s1"> "name": "dom1"</span>
<span class="s1"> }</span>
<span class="s1">}'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.domain.id<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'"'</span><span class="k">)</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"ID of dom1: </span><span class="nv">$ID_DOM1</span><span class="s2">"</span>
<span class="nv">ID_DOM2</span><span class="o">=</span><span class="k">$(</span><span class="se">\</span>
curl<span class="w"> </span>http://localhost:5000/v3/domains<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$CLOUD_ADMIN_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type: application/json"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">'</span>
<span class="s1">{</span>
<span class="s1"> "domain": {</span>
<span class="s1"> "enabled": true,</span>
<span class="s1"> "name": "dom2"</span>
<span class="s1"> }</span>
<span class="s1">}'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.domain.id<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'"'</span><span class="k">)</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"ID of dom2: </span><span class="nv">$ID_DOM2</span><span class="s2">"</span>
</code></pre></div>
</li>
<li>
<p>Now we will create a user <code>adm1</code> in domain <code>dom1</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nv">ID_ADM1</span><span class="o">=</span><span class="k">$(</span><span class="se">\</span>
curl<span class="w"> </span>http://localhost:5000/v3/users<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$CLOUD_ADMIN_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type: application/json"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span>
<span class="s2">{</span>
<span class="s2"> \"user\": {</span>
<span class="s2"> \"description\": \"Administrator of domain dom1\",</span>
<span class="s2"> \"domain_id\": \"</span><span class="nv">$ID_DOM1</span><span class="s2">\",</span>
<span class="s2"> \"enabled\": true,</span>
<span class="s2"> \"name\": \"adm1\",</span>
<span class="s2"> \"password\": \"password\"</span>
<span class="s2"> }</span>
<span class="s2">}"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.user.id<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'"'</span><span class="k">)</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"ID of user adm1: </span><span class="nv">$ID_ADM1</span><span class="s2">"</span>
</code></pre></div>
</li>
<li>
<p>We will also grant the <code>admin</code> role on domain <code>dom1</code> to this <code>adm1</code>
user.</p>
<div class="highlight"><pre><span></span><code>curl<span class="w"> </span>-X<span class="w"> </span>PUT<span class="w"> </span>http://localhost:5000/v3/domains/<span class="si">${</span><span class="nv">ID_DOM1</span><span class="si">}</span>/users/<span class="si">${</span><span class="nv">ID_ADM1</span><span class="si">}</span>/roles/<span class="si">${</span><span class="nv">ADMIN_ROLE_ID</span><span class="si">}</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-i<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$CLOUD_ADMIN_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type: application/json"</span>
curl<span class="w"> </span>http://localhost:5000/v3/domains/<span class="si">${</span><span class="nv">ID_DOM1</span><span class="si">}</span>/users/<span class="si">${</span><span class="nv">ID_ADM1</span><span class="si">}</span>/roles<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$CLOUD_ADMIN_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.roles
</code></pre></div>
</li>
</ul>
<h2>Creating projects and users</h2>
<p>The <code>adm1</code> user can now fully manage domain <code>dom1</code>. He is allowed to
manage as many projects and users as he wishes within <code>dom1</code>, while
not being able to access resources of domain <code>dom2</code>.</p>
<ul>
<li>
<p>Now we authenticate as user <code>adm1</code> with a scope on <code>dom1</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nv">ADM1_TOKEN</span><span class="o">=</span><span class="k">$(</span><span class="se">\</span>
curl<span class="w"> </span>http://localhost:5000/v3/auth/tokens<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-i<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type: application/json"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">'</span>
<span class="s1">{</span>
<span class="s1"> "auth": {</span>
<span class="s1"> "identity": {</span>
<span class="s1"> "methods": [</span>
<span class="s1"> "password"</span>
<span class="s1"> ],</span>
<span class="s1"> "password": {</span>
<span class="s1"> "user": {</span>
<span class="s1"> "domain": {</span>
<span class="s1"> "name": "dom1"</span>
<span class="s1"> },</span>
<span class="s1"> "name": "adm1",</span>
<span class="s1"> "password": "password"</span>
<span class="s1"> }</span>
<span class="s1"> }</span>
<span class="s1"> },</span>
<span class="s1"> "scope": {</span>
<span class="s1"> "domain": {</span>
<span class="s1"> "name": "dom1"</span>
<span class="s1"> }</span>
<span class="s1"> }</span>
<span class="s1"> }</span>
<span class="s1">}'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>^X-Subject-Token:<span class="w"> </span><span class="p">|</span><span class="w"> </span>awk<span class="w"> </span><span class="s1">'{print $2}'</span><span class="w"> </span><span class="k">)</span>
</code></pre></div>
</li>
<li>
<p>We create a project <code>prj1</code> in domain <code>dom1</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nv">ID_PRJ1</span><span class="o">=</span><span class="k">$(</span><span class="se">\</span>
curl<span class="w"> </span>http://localhost:5000/v3/projects<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$ADM1_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type: application/json"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span>
<span class="s2">{</span>
<span class="s2"> \"project\": {</span>
<span class="s2"> \"enabled\": true,</span>
<span class="s2"> \"domain_id\": \"</span><span class="nv">$ID_DOM1</span><span class="s2">\",</span>
<span class="s2"> \"name\": \"prj1\"</span>
<span class="s2"> }\</span>
<span class="s2">}"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.project.id<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'"'</span><span class="w"> </span><span class="k">)</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"ID of prj1: </span><span class="nv">$ID_PRJ1</span><span class="s2">"</span>
</code></pre></div>
</li>
<li>
<p>When trying and creating a project in domain <code>dom2</code>, it fails.</p>
<div class="highlight"><pre><span></span><code>curl<span class="w"> </span>http://localhost:5000/v3/projects<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$ADM1_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type: application/json"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span>
<span class="s2">{</span>
<span class="s2"> \"project\": {</span>
<span class="s2"> \"enabled\": true,</span>
<span class="s2"> \"domain_id\": \"</span><span class="nv">$ID_DOM2</span><span class="s2">\",</span>
<span class="s2"> \"name\": \"prj2\"</span>
<span class="s2"> }\</span>
<span class="s2">}"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.
</code></pre></div>
</li>
<li>
<p>Creating a standard user <code>usr1</code> in domain <code>dom1</code>, with default project <code>prj1</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nv">ID_USR1</span><span class="o">=</span><span class="k">$(</span><span class="se">\</span>
curl<span class="w"> </span>http://localhost:5000/v3/users<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$ADM1_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type: application/json"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s2">"</span>
<span class="s2">{</span>
<span class="s2"> \"user\": {</span>
<span class="s2"> \"default_project_id\": \"</span><span class="nv">$ID_PRJ1</span><span class="s2">\",</span>
<span class="s2"> \"description\": \"Just a user of dom1\",</span>
<span class="s2"> \"domain_id\": \"</span><span class="nv">$ID_DOM1</span><span class="s2">\",</span>
<span class="s2"> \"enabled\": true,</span>
<span class="s2"> \"name\": \"usr1\",</span>
<span class="s2"> \"password\": \"password\"</span>
<span class="s2"> }</span>
<span class="s2">}"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.user.id<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'"'</span><span class="w"> </span><span class="k">)</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"ID of user usr1: </span><span class="nv">$ID_USR1</span><span class="s2">"</span>
</code></pre></div>
</li>
<li>
<p>Granting <code>Member</code> role to user <code>usr1</code> on project <code>prj1</code>.</p>
<div class="highlight"><pre><span></span><code><span class="nv">MEMBER_ROLE_ID</span><span class="o">=</span><span class="k">$(</span><span class="se">\</span>
curl<span class="w"> </span>http://localhost:5000/v3/roles?name<span class="o">=</span>Member<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$ADM1_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.roles<span class="o">[</span><span class="m">0</span><span class="o">]</span>.id<span class="w"> </span><span class="p">|</span><span class="w"> </span>tr<span class="w"> </span>-d<span class="w"> </span><span class="s1">'"'</span><span class="w"> </span><span class="k">)</span>
curl<span class="w"> </span>-X<span class="w"> </span>PUT<span class="w"> </span>http://localhost:5000/v3/projects/<span class="si">${</span><span class="nv">ID_PRJ1</span><span class="si">}</span>/users/<span class="si">${</span><span class="nv">ID_USR1</span><span class="si">}</span>/roles/<span class="si">${</span><span class="nv">MEMBER_ROLE_ID</span><span class="si">}</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-i<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$ADM1_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"Content-Type: application/json"</span>
curl<span class="w"> </span>http://localhost:5000/v3/projects/<span class="si">${</span><span class="nv">ID_PRJ1</span><span class="si">}</span>/users/<span class="si">${</span><span class="nv">ID_USR1</span><span class="si">}</span>/roles<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-s<span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">"X-Auth-Token: </span><span class="nv">$ADM1_TOKEN</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span>.roles
</code></pre></div>
</li>
</ul>
<p>The domain administrator <code>adm1</code> ended up creating a project <code>prj1</code> and
a user <code>usr1</code> member of the project. <code>usr1</code> can now get a token scoped
on <code>prj1</code> and manage resources into this project.</p>Openstack volume in-use although VM doesn't exist2014-01-04T00:00:00+01:002014-01-04T00:00:00+01:00Florent Flamenttag:www.florentflament.com,2014-01-04:/blog/openstack-volume-in-use-although-vm-doesnt-exist.html<p>A user at <a href="http://www.cloudwatt.com">Cloudwatt</a> had an issue about Cinder volumes with
status <code>in-use</code>, attached to VMs that didn't exist anymore. I could
find similar bugs referenced in launchpad <a href="https://bugs.launchpad.net/cinder/+bug/1201418">here</a> and <a href="https://bugs.launchpad.net/nova/+bug/1096197">there</a>,
both with status <code>invalid</code>... But I didn't succeed in reproducing the
bug (using both the Horizon dashboard and the …</p><p>A user at <a href="http://www.cloudwatt.com">Cloudwatt</a> had an issue about Cinder volumes with
status <code>in-use</code>, attached to VMs that didn't exist anymore. I could
find similar bugs referenced in launchpad <a href="https://bugs.launchpad.net/cinder/+bug/1201418">here</a> and <a href="https://bugs.launchpad.net/nova/+bug/1096197">there</a>,
both with status <code>invalid</code>... But I didn't succeed in reproducing the
bug (using both the Horizon dashboard and the CLIs) until I got
feedback about what our user was doing.</p>
<h2>The issue</h2>
<p>The issue appears when one tries to attach a volume by using the
<code>python-cinderclient</code> library in some Python code. There are actually
some weird methods for both version of the API.</p>
<ul>
<li><a href="https://github.com/openstack/python-cinderclient/blob/master/cinderclient/v1/volumes.py">cinderclient.v1.volumes.VolumeManager.attach</a></li>
<li><a href="https://github.com/openstack/python-cinderclient/blob/master/cinderclient/v1/volumes.py">cinderclient.v2.volumes.VolumeManager.attach</a></li>
</ul>
<p>These methods call the Cinder <code>POST /volumes/{volume_id}/action</code> API,
which is neither documented <a href="http://docs.openstack.org/api/openstack-block-storage/2.0/content/Volumes.html">here</a> nor <a href="http://api.openstack.org/api-ref-blockstorage.html">there</a>. They can be used
to "set attachment metadata", which according to my opinion has no
good reason to be performed directly by a user.</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">time</span>
<span class="o">>>></span> <span class="kn">import</span> <span class="nn">cinderclient.v1.client</span>
<span class="o">>>></span> <span class="kn">import</span> <span class="nn">novaclient.v1_1.client</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="c1"># Creating manager</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">args</span> <span class="o">=</span> <span class="p">[</span><span class="s2">"username"</span><span class="p">,</span> <span class="s2">"password"</span><span class="p">,</span> <span class="s2">"project_name"</span><span class="p">,</span> <span class="s2">"auth_url"</span><span class="p">]</span>
<span class="o">>>></span> <span class="n">cinder</span> <span class="o">=</span> <span class="n">cinderclient</span><span class="o">.</span><span class="n">v1</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">Client</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">nova</span> <span class="o">=</span> <span class="n">novaclient</span><span class="o">.</span><span class="n">v1_1</span><span class="o">.</span><span class="n">client</span><span class="o">.</span><span class="n">Client</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">)</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="c1"># Getting an image and a flavor to launch our VM</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">img</span> <span class="o">=</span> <span class="n">nova</span><span class="o">.</span><span class="n">images</span><span class="o">.</span><span class="n">list</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
<span class="o">>>></span> <span class="n">fla</span> <span class="o">=</span> <span class="n">nova</span><span class="o">.</span><span class="n">flavors</span><span class="o">.</span><span class="n">list</span><span class="p">()[</span><span class="mi">0</span><span class="p">]</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="c1"># Creating our resources (vm and volume)</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">vm</span> <span class="o">=</span> <span class="n">nova</span><span class="o">.</span><span class="n">servers</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="s2">"MyVm"</span><span class="p">,</span> <span class="n">img</span><span class="p">,</span> <span class="n">fla</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">vol</span> <span class="o">=</span> <span class="n">cinder</span><span class="o">.</span><span class="n">volumes</span><span class="o">.</span><span class="n">create</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="n">display_name</span><span class="o">=</span><span class="s2">"MyVol"</span><span class="p">)</span>
<span class="o">>>></span>
<span class="o">>>></span> <span class="c1"># Waiting for VM to be spawned</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="k">while</span> <span class="n">nova</span><span class="o">.</span><span class="n">servers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">vm</span><span class="o">.</span><span class="n">id</span><span class="p">)</span><span class="o">.</span><span class="n">status</span> <span class="o">!=</span> <span class="s2">"ACTIVE"</span><span class="p">:</span>
<span class="o">...</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="c1"># Try an attach the volume using the VolumeManager.attach method</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">cinder</span><span class="o">.</span><span class="n">volumes</span><span class="o">.</span><span class="n">attach</span><span class="p">(</span><span class="n">vol</span><span class="p">,</span> <span class="n">vm</span><span class="o">.</span><span class="n">id</span><span class="p">,</span> <span class="s2">"/dev/vdb"</span><span class="p">)</span>
<span class="p">(</span><span class="o"><</span><span class="n">Response</span> <span class="p">[</span><span class="mi">202</span><span class="p">]</span><span class="o">></span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">cinder</span><span class="o">.</span><span class="n">volumes</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">vol</span><span class="o">.</span><span class="n">id</span><span class="p">)</span><span class="o">.</span><span class="n">_info</span><span class="p">[</span><span class="s2">"attachments"</span><span class="p">]</span>
<span class="p">[{</span><span class="sa">u</span><span class="s1">'device'</span><span class="p">:</span> <span class="sa">u</span><span class="s1">'/dev/vdb'</span><span class="p">,</span> <span class="sa">u</span><span class="s1">'server_id'</span><span class="p">:</span> <span class="sa">u</span><span class="s1">'3ee03547-7b93-4c8f-9316-bc2adafbd08a'</span><span class="p">,</span> <span class="sa">u</span><span class="s1">'volume_id'</span><span class="p">:</span> <span class="sa">u</span><span class="s1">'f024883d-4b35-4894-9fbf-51498e6c3c09'</span><span class="p">,</span> <span class="sa">u</span><span class="s1">'host_name'</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span> <span class="sa">u</span><span class="s1">'id'</span><span class="p">:</span> <span class="sa">u</span><span class="s1">'f024883d-4b35-4894-9fbf-51498e6c3c09'</span><span class="p">}]</span>
<span class="o">>>></span> <span class="c1"># Well it looks like it is attached</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">vm</span> <span class="o">=</span> <span class="n">nova</span><span class="o">.</span><span class="n">servers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'3ee03547-7b93-4c8f-9316-bc2adafbd08a'</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">vm</span><span class="o">.</span><span class="n">_info</span><span class="p">[</span><span class="s1">'os-extended-volumes:volumes_attached'</span><span class="p">]</span>
<span class="p">[]</span>
<span class="o">>>></span> <span class="c1"># But actually it isn't</span>
<span class="o">...</span>
<span class="o">>>></span>
</code></pre></div>
<h2>The solution</h2>
<p>The solution is to use the appropriate method to attach a volume to an
instance. This should be done by using the
<a href="https://github.com/openstack/python-novaclient/blob/master/novaclient/v1_1/volumes.py">novaclient.v1_1.volumes.VolumeManager.create_server_volume</a>
method. It actually allows to "attach a volume to a server", which is
what we want to do (And yes the method's name is not super clear).</p>
<div class="highlight"><pre><span></span><code><span class="o">>>></span> <span class="c1"># Let's continue on the example above</span>
<span class="o">...</span> <span class="c1"># First by clearing our volume attachment's metadata</span>
<span class="o">...</span> <span class="c1"># to recover a consistent state</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">vol</span><span class="o">.</span><span class="n">detach</span><span class="p">()</span>
<span class="p">(</span><span class="o"><</span><span class="n">Response</span> <span class="p">[</span><span class="mi">202</span><span class="p">]</span><span class="o">></span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">cinder</span><span class="o">.</span><span class="n">volumes</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">vol</span><span class="o">.</span><span class="n">id</span><span class="p">)</span><span class="o">.</span><span class="n">_info</span><span class="p">[</span><span class="s1">'attachments'</span><span class="p">]</span>
<span class="p">[]</span>
<span class="o">>>></span> <span class="n">nova</span><span class="o">.</span><span class="n">servers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">vm</span><span class="o">.</span><span class="n">id</span><span class="p">)</span><span class="o">.</span><span class="n">_info</span><span class="p">[</span><span class="s1">'os-extended-volumes:volumes_attached'</span><span class="p">]</span>
<span class="p">[]</span>
<span class="o">>>></span> <span class="c1"># Now we can use the VolumeManager.create_server_volume method</span>
<span class="o">...</span> <span class="c1"># to really attach our volume to our vm</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">nova</span><span class="o">.</span><span class="n">volumes</span><span class="o">.</span><span class="n">create_server_volume</span><span class="p">(</span><span class="n">vm</span><span class="o">.</span><span class="n">id</span><span class="p">,</span> <span class="n">vol</span><span class="o">.</span><span class="n">id</span><span class="p">,</span> <span class="s2">"/dev/vdb"</span><span class="p">)</span>
<span class="o"><</span><span class="n">Volume</span><span class="p">:</span> <span class="n">f024883d</span><span class="o">-</span><span class="mi">4</span><span class="n">b35</span><span class="o">-</span><span class="mi">4894</span><span class="o">-</span><span class="mi">9</span><span class="n">fbf</span><span class="o">-</span><span class="mf">51498e6</span><span class="n">c3c09</span><span class="o">></span>
<span class="o">>>></span> <span class="n">cinder</span><span class="o">.</span><span class="n">volumes</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">vol</span><span class="o">.</span><span class="n">id</span><span class="p">)</span><span class="o">.</span><span class="n">_info</span><span class="p">[</span><span class="s2">"attachments"</span><span class="p">]</span>
<span class="p">[{</span><span class="sa">u</span><span class="s1">'device'</span><span class="p">:</span> <span class="sa">u</span><span class="s1">'/dev/vdb'</span><span class="p">,</span> <span class="sa">u</span><span class="s1">'server_id'</span><span class="p">:</span> <span class="sa">u</span><span class="s1">'3ee03547-7b93-4c8f-9316-bc2adafbd08a'</span><span class="p">,</span> <span class="sa">u</span><span class="s1">'volume_id'</span><span class="p">:</span> <span class="sa">u</span><span class="s1">'f024883d-4b35-4894-9fbf-51498e6c3c09'</span><span class="p">,</span> <span class="sa">u</span><span class="s1">'host_name'</span><span class="p">:</span> <span class="kc">None</span><span class="p">,</span> <span class="sa">u</span><span class="s1">'id'</span><span class="p">:</span> <span class="sa">u</span><span class="s1">'f024883d-4b35-4894-9fbf-51498e6c3c09'</span><span class="p">}]</span>
<span class="o">>>></span> <span class="n">nova</span><span class="o">.</span><span class="n">servers</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">vm</span><span class="o">.</span><span class="n">id</span><span class="p">)</span><span class="o">.</span><span class="n">_info</span><span class="p">[</span><span class="s1">'os-extended-volumes:volumes_attached'</span><span class="p">]</span>
<span class="p">[{</span><span class="sa">u</span><span class="s1">'id'</span><span class="p">:</span> <span class="sa">u</span><span class="s1">'f024883d-4b35-4894-9fbf-51498e6c3c09'</span><span class="p">}]</span>
<span class="o">>>></span>
</code></pre></div>
<h2>The bug</h2>
<p>Although, we could argue that the inconsistent state is due to a bad
usage of the client libraries, I do believe that Cinder API (and
therefore python-cinderclient) should not allow the user to put its
resources in an inconsistent state. Moreover, it isn't possible to
recover from such state by using the Horizon dashboard or the CLIs. To
do so, one has to either use the Python client libraries, or update
entries in the database manually...</p>
<h2>Recovering from inconsistent state</h2>
<p>One way to recover from a "volume attached to non-existent VM"
inconsistent state, is to manually update entries in the Cinder
database.</p>
<div class="highlight"><pre><span></span><code>$<span class="w"> </span>mysql<span class="w"> </span>cinder
Reading<span class="w"> </span>table<span class="w"> </span>information<span class="w"> </span><span class="k">for</span><span class="w"> </span>completion<span class="w"> </span>of<span class="w"> </span>table<span class="w"> </span>and<span class="w"> </span>column<span class="w"> </span>names
You<span class="w"> </span>can<span class="w"> </span>turn<span class="w"> </span>off<span class="w"> </span>this<span class="w"> </span>feature<span class="w"> </span>to<span class="w"> </span>get<span class="w"> </span>a<span class="w"> </span>quicker<span class="w"> </span>startup<span class="w"> </span>with<span class="w"> </span>-A
Welcome<span class="w"> </span>to<span class="w"> </span>the<span class="w"> </span>MySQL<span class="w"> </span>monitor.<span class="w"> </span>Commands<span class="w"> </span>end<span class="w"> </span>with<span class="w"> </span><span class="p">;</span><span class="w"> </span>or<span class="w"> </span><span class="se">\g</span>.
Your<span class="w"> </span>MySQL<span class="w"> </span>connection<span class="w"> </span>id<span class="w"> </span>is<span class="w"> </span><span class="m">1900</span>
Server<span class="w"> </span>version:<span class="w"> </span><span class="m">5</span>.5.34-0ubuntu0.12.04.1-log<span class="w"> </span><span class="o">(</span>Ubuntu<span class="o">)</span>
Copyright<span class="w"> </span><span class="o">(</span>c<span class="o">)</span><span class="w"> </span><span class="m">2000</span>,<span class="w"> </span><span class="m">2013</span>,<span class="w"> </span>Oracle<span class="w"> </span>and/or<span class="w"> </span>its<span class="w"> </span>affiliates.<span class="w"> </span>All<span class="w"> </span>rights<span class="w"> </span>reserved.
Oracle<span class="w"> </span>is<span class="w"> </span>a<span class="w"> </span>registered<span class="w"> </span>trademark<span class="w"> </span>of<span class="w"> </span>Oracle<span class="w"> </span>Corporation<span class="w"> </span>and/or<span class="w"> </span>its
affiliates.<span class="w"> </span>Other<span class="w"> </span>names<span class="w"> </span>may<span class="w"> </span>be<span class="w"> </span>trademarks<span class="w"> </span>of<span class="w"> </span>their<span class="w"> </span>respective
owners.
Type<span class="w"> </span><span class="s1">'help;'</span><span class="w"> </span>or<span class="w"> </span><span class="s1">'\h'</span><span class="w"> </span><span class="k">for</span><span class="w"> </span>help.<span class="w"> </span>Type<span class="w"> </span><span class="s1">'\c'</span><span class="w"> </span>to<span class="w"> </span>clear<span class="w"> </span>the<span class="w"> </span>current<span class="w"> </span>input<span class="w"> </span>statement.
mysql><span class="w"> </span>SELECT<span class="w"> </span>id,status,attach_status,mountpoint,instance_uuid<span class="w"> </span>from<span class="w"> </span>volumes<span class="p">;</span>
+--------------------------------------+---------+---------------+------------+--------------------------------------+
<span class="p">|</span><span class="w"> </span>id<span class="w"> </span><span class="p">|</span><span class="w"> </span>status<span class="w"> </span><span class="p">|</span><span class="w"> </span>attach_status<span class="w"> </span><span class="p">|</span><span class="w"> </span>mountpoint<span class="w"> </span><span class="p">|</span><span class="w"> </span>instance_uuid<span class="w"> </span><span class="p">|</span>
+--------------------------------------+---------+---------------+------------+--------------------------------------+
<span class="p">|</span><span class="w"> </span>0580142b-bfb5-4113-8676-4fb783ec05f2<span class="w"> </span><span class="p">|</span><span class="w"> </span>deleted<span class="w"> </span><span class="p">|</span><span class="w"> </span>detached<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>1085d96e-ae82-484b-8495-27cc2f25c9c3<span class="w"> </span><span class="p">|</span><span class="w"> </span>deleted<span class="w"> </span><span class="p">|</span><span class="w"> </span>detached<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>4fbff7ce-2c9f-4116-ad1e-12f78001da2d<span class="w"> </span><span class="p">|</span><span class="w"> </span>deleted<span class="w"> </span><span class="p">|</span><span class="w"> </span>detached<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>57b32eaf-7b71-49bf-a8fd-4115567a6cda<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="k">in</span>-use<span class="w"> </span><span class="p">|</span><span class="w"> </span>attached<span class="w"> </span><span class="p">|</span><span class="w"> </span>/dev/vdb<span class="w"> </span><span class="p">|</span><span class="w"> </span>fa53d190-426b-4ce6-8d36-1af408c25225<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>60a7eb30-9849-4b8d-9ca1-8f554b9a4045<span class="w"> </span><span class="p">|</span><span class="w"> </span>deleted<span class="w"> </span><span class="p">|</span><span class="w"> </span>detached<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>a34ca08a-2e6b-4820-b472-91aa27b47261<span class="w"> </span><span class="p">|</span><span class="w"> </span>deleted<span class="w"> </span><span class="p">|</span><span class="w"> </span>detached<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>d827daa5-c8d7-427b-a53c-8be918a1a6fb<span class="w"> </span><span class="p">|</span><span class="w"> </span>deleted<span class="w"> </span><span class="p">|</span><span class="w"> </span>detached<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span>
+--------------------------------------+---------+---------------+------------+--------------------------------------+
<span class="m">7</span><span class="w"> </span>rows<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">(</span><span class="m">0</span>.00<span class="w"> </span>sec<span class="o">)</span>
mysql><span class="w"> </span>UPDATE<span class="w"> </span>volumes<span class="w"> </span>SET<span class="w"> </span><span class="nv">status</span><span class="o">=</span><span class="s2">"available"</span>,<span class="w"> </span><span class="nv">attach_status</span><span class="o">=</span><span class="s2">"detached"</span>,<span class="w"> </span><span class="nv">mountpoint</span><span class="o">=</span>NULL,<span class="w"> </span><span class="nv">instance_uuid</span><span class="o">=</span>NULL<span class="w"> </span>WHERE<span class="w"> </span><span class="nv">id</span><span class="o">=</span><span class="s2">"57b32eaf-7b71-49bf-a8fd-4115567a6cda"</span><span class="p">;</span>
Query<span class="w"> </span>OK,<span class="w"> </span><span class="m">1</span><span class="w"> </span>row<span class="w"> </span>affected<span class="w"> </span><span class="o">(</span><span class="m">0</span>.00<span class="w"> </span>sec<span class="o">)</span>
Rows<span class="w"> </span>matched:<span class="w"> </span><span class="m">1</span><span class="w"> </span>Changed:<span class="w"> </span><span class="m">1</span><span class="w"> </span>Warnings:<span class="w"> </span><span class="m">0</span>
mysql><span class="w"> </span>SELECT<span class="w"> </span>id,status,attach_status,mountpoint,instance_uuid<span class="w"> </span>from<span class="w"> </span>volumes<span class="p">;</span>
+--------------------------------------+-----------+---------------+------------+---------------+
<span class="p">|</span><span class="w"> </span>id<span class="w"> </span><span class="p">|</span><span class="w"> </span>status<span class="w"> </span><span class="p">|</span><span class="w"> </span>attach_status<span class="w"> </span><span class="p">|</span><span class="w"> </span>mountpoint<span class="w"> </span><span class="p">|</span><span class="w"> </span>instance_uuid<span class="w"> </span><span class="p">|</span>
+--------------------------------------+-----------+---------------+------------+---------------+
<span class="p">|</span><span class="w"> </span>0580142b-bfb5-4113-8676-4fb783ec05f2<span class="w"> </span><span class="p">|</span><span class="w"> </span>deleted<span class="w"> </span><span class="p">|</span><span class="w"> </span>detached<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>1085d96e-ae82-484b-8495-27cc2f25c9c3<span class="w"> </span><span class="p">|</span><span class="w"> </span>deleted<span class="w"> </span><span class="p">|</span><span class="w"> </span>detached<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>4fbff7ce-2c9f-4116-ad1e-12f78001da2d<span class="w"> </span><span class="p">|</span><span class="w"> </span>deleted<span class="w"> </span><span class="p">|</span><span class="w"> </span>detached<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>57b32eaf-7b71-49bf-a8fd-4115567a6cda<span class="w"> </span><span class="p">|</span><span class="w"> </span>available<span class="w"> </span><span class="p">|</span><span class="w"> </span>detached<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>60a7eb30-9849-4b8d-9ca1-8f554b9a4045<span class="w"> </span><span class="p">|</span><span class="w"> </span>deleted<span class="w"> </span><span class="p">|</span><span class="w"> </span>detached<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>a34ca08a-2e6b-4820-b472-91aa27b47261<span class="w"> </span><span class="p">|</span><span class="w"> </span>deleted<span class="w"> </span><span class="p">|</span><span class="w"> </span>detached<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span>
<span class="p">|</span><span class="w"> </span>d827daa5-c8d7-427b-a53c-8be918a1a6fb<span class="w"> </span><span class="p">|</span><span class="w"> </span>deleted<span class="w"> </span><span class="p">|</span><span class="w"> </span>detached<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span><span class="w"> </span>NULL<span class="w"> </span><span class="p">|</span>
+--------------------------------------+-----------+---------------+------------+---------------+
<span class="m">7</span><span class="w"> </span>rows<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="o">(</span><span class="m">0</span>.00<span class="w"> </span>sec<span class="o">)</span>
mysql><span class="w"> </span>Bye
$<span class="w"> </span>cinder<span class="w"> </span>list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
<span class="p">|</span><span class="w"> </span>ID<span class="w"> </span><span class="p">|</span><span class="w"> </span>Status<span class="w"> </span><span class="p">|</span><span class="w"> </span>Name<span class="w"> </span><span class="p">|</span><span class="w"> </span>Size<span class="w"> </span><span class="p">|</span><span class="w"> </span>Volume<span class="w"> </span>Type<span class="w"> </span><span class="p">|</span><span class="w"> </span>Bootable<span class="w"> </span><span class="p">|</span><span class="w"> </span>Attached<span class="w"> </span>to<span class="w"> </span><span class="p">|</span>
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
<span class="p">|</span><span class="w"> </span>57b32eaf-7b71-49bf-a8fd-4115567a6cda<span class="w"> </span><span class="p">|</span><span class="w"> </span>available<span class="w"> </span><span class="p">|</span><span class="w"> </span>vol1<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>None<span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="nb">false</span><span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="p">|</span>
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
</code></pre></div>Ceph and Cinder multi-backend2013-11-17T00:00:00+01:002013-11-17T00:00:00+01:00Florent Flamenttag:www.florentflament.com,2013-11-17:/blog/ceph-and-cinder-multi-backend.html<p><a href="http://ceph.com/docs/master/">Ceph's documentation</a> is quite extensive. However, when trying and
<a href="http://ceph.com/docs/master/rbd/rbd-openstack/">installing Ceph on a running Openstack platform</a>, I met two main
issues: How to deal with a multi-backend setup? And how to deal with
several nova-compute nodes? This note will focus on the steps that I
followed in order to have …</p><p><a href="http://ceph.com/docs/master/">Ceph's documentation</a> is quite extensive. However, when trying and
<a href="http://ceph.com/docs/master/rbd/rbd-openstack/">installing Ceph on a running Openstack platform</a>, I met two main
issues: How to deal with a multi-backend setup? And how to deal with
several nova-compute nodes? This note will focus on the steps that I
followed in order to have Ceph running as a Cinder backend (among
other backends), using <a href="http://ceph.com/docs/master/rados/operations/authentication/">cephx authentication</a>.</p>
<h2>Ceph node</h2>
<p>As described on Ceph's documentation, one has to create a pool on the
Ceph nodes (Ceph's doc provides extensive <a href="http://ceph.com/docs/master/rados/operations/placement-groups/">documentation about the
number of placement groups that should be used</a>). The following
command has to be launched on any Ceph node:</p>
<div class="highlight"><pre><span></span><code>ceph<span class="w"> </span>osd<span class="w"> </span>pool<span class="w"> </span>create<span class="w"> </span>volumes<span class="w"> </span><span class="m">128</span>
</code></pre></div>
<p>Because of cephx authentication, we have to create a new user with the
appropriate rights for cinder and nova to be able to access Ceph's
storage. The following command has to be launched on a Ceph node:</p>
<div class="highlight"><pre><span></span><code>ceph<span class="w"> </span>auth<span class="w"> </span>get-or-create<span class="w"> </span>client.volumes<span class="w"> </span>mon<span class="w"> </span><span class="s1">'allow r'</span><span class="w"> </span>osd<span class="w"> </span><span class="s1">'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images'</span>
</code></pre></div>
<p>The keyring (token used to identify Ceph users) has to be copied on
cinder-volume nodes. The keyring file can be created with the
following command on a Ceph node:</p>
<div class="highlight"><pre><span></span><code>ceph<span class="w"> </span>auth<span class="w"> </span>get-or-create<span class="w"> </span>client.volumes<span class="w"> </span>><span class="w"> </span>ceph.client.images.keyring
</code></pre></div>
<h2>Cinder-volume nodes</h2>
<p>The file created above has to be copied on the cinder-volume nodes, in
the directory <code>/etc/ceph</code> with the propoer uid and gid:</p>
<div class="highlight"><pre><span></span><code>chown<span class="w"> </span>cinder:cinder<span class="w"> </span>/etc/ceph/ceph.client.images.keyring
</code></pre></div>
<p>Ceph's configuration file <code>/etc/ceph/ceph.conf</code> has to be copied at
the same location.</p>
<p>Then we have to install the following packages on these nodes:</p>
<div class="highlight"><pre><span></span><code>sudo<span class="w"> </span>apt-get<span class="w"> </span>install<span class="w"> </span>python-ceph<span class="w"> </span>ceph-common
</code></pre></div>
<p>Cinder configuration file <code>/etc/cinder/cinder.conf</code> has to be updated,
by <a href="https://wiki.openstack.org/wiki/Cinder-multi-backend">setting a new backend</a>. A new backend that we'll call <code>ceph</code>
will be added to the <code>enabled_backends</code> parameter, and the
corresponding backend section will be created:</p>
<div class="highlight"><pre><span></span><code><span class="n">enabled_backends</span> <span class="o">=</span> <span class="n">former</span><span class="o">-</span><span class="n">backend</span><span class="p">,</span><span class="n">ceph</span>
<span class="p">[</span><span class="n">former</span><span class="o">-</span><span class="n">backend</span><span class="p">]</span>
<span class="o">...</span>
<span class="p">[</span><span class="n">ceph</span><span class="p">]</span>
<span class="n">volume_driver</span> <span class="o">=</span> <span class="n">cinder</span><span class="o">.</span><span class="n">volume</span><span class="o">.</span><span class="n">drivers</span><span class="o">.</span><span class="n">rbd</span><span class="o">.</span><span class="n">RBDDriver</span>
<span class="n">volume_backend_name</span> <span class="o">=</span> <span class="n">ceph</span>
<span class="n">rbd_pool</span> <span class="o">=</span> <span class="n">volumes</span>
<span class="n">glance_api_version</span> <span class="o">=</span> <span class="mi">2</span>
<span class="n">rbd_user</span> <span class="o">=</span> <span class="n">volumes</span>
<span class="n">rbd_secret_uuid</span> <span class="o">=</span> <span class="n">uuid_of_secret</span>
</code></pre></div>
<p>The <code>rbd_secret_uuid</code> value cannot be set right now, this parameter
will allow nova to mount Ceph block devices. We will update this value
in a next step.</p>
<p>If the <code>scheduler_driver</code> parameter is not set to FilterScheduler, it
has to be updated:</p>
<div class="highlight"><pre><span></span><code><span class="n">scheduler_driver</span> <span class="o">=</span> <span class="n">cinder</span><span class="o">.</span><span class="n">scheduler</span><span class="o">.</span><span class="n">filter_scheduler</span><span class="o">.</span><span class="n">FilterScheduler</span>
</code></pre></div>
<p>Once the configuration file updated, cinder-volume service has to be
restarted to load the new configuration:</p>
<div class="highlight"><pre><span></span><code>sudo<span class="w"> </span>service<span class="w"> </span>cinder-volume<span class="w"> </span>restart
</code></pre></div>
<p>And a new volume-type has to be added to Cinder, with the following
command, which has to be called with an adminitrator credentials
(OS_USERNAME, OS_TENANT_NAME and OS_PASSWORD):</p>
<div class="highlight"><pre><span></span><code>cinder<span class="w"> </span>type-create<span class="w"> </span>ceph
cinder<span class="w"> </span>type-key<span class="w"> </span>ceph<span class="w"> </span><span class="nb">set</span><span class="w"> </span><span class="nv">volume_backend_name</span><span class="o">=</span>ceph
</code></pre></div>
<p>At that point, we should be able to create new Cinder volumes using
Ceph as a backend, with the following command:</p>
<div class="highlight"><pre><span></span><code>cinder<span class="w"> </span>create<span class="w"> </span>--volume-type<span class="w"> </span>ceph<span class="w"> </span>--display-name<span class="w"> </span>ceph-test<span class="w"> </span><span class="m">1</span>
cinder<span class="w"> </span>list
</code></pre></div>
<h2>Nova-compute nodes</h2>
<p>Now we have to configure our nova-compute nodes to allow our VMs to
mount Ceph block devices. To do that, we have to dump Ceph's
authentication token to a file that we'll use on each nova-compute
node. On a Ceph node:</p>
<div class="highlight"><pre><span></span><code>ceph<span class="w"> </span>auth<span class="w"> </span>get-key<span class="w"> </span>client.volumes<span class="w"> </span>><span class="w"> </span>client.volumes.key
</code></pre></div>
<p>We will also need a <code>secret.xml</code> file that will be used on each
compute node, with the following initial content:</p>
<div class="highlight"><pre><span></span><code><span class="nt"><secret</span><span class="w"> </span><span class="na">ephemeral=</span><span class="s">'no'</span><span class="w"> </span><span class="na">private=</span><span class="s">'no'</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><usage</span><span class="w"> </span><span class="na">type=</span><span class="s">'ceph'</span><span class="nt">></span>
<span class="w"> </span><span class="nt"><name></span>client.volumes<span class="w"> </span>secret<span class="nt"></name></span>
<span class="w"> </span><span class="nt"></usage></span>
<span class="nt"></secret></span>
</code></pre></div>
<p>Now we can copy these two files (<code>client.volumes.key</code> and
<code>secret.xml</code>) on any nova-compute node. We'll call this node our first
nova-compute node. On this first node we will define a secret with the
following command:</p>
<div class="highlight"><pre><span></span><code>virsh<span class="w"> </span>secret-define<span class="w"> </span>--file<span class="w"> </span>secret.xml
</code></pre></div>
<p>The UUID_OF_SECRET displayed has to be copied somewhere, since it will
be used multiple times to configure nova-compute, as well as
cinder-volume. We can then update the secret's value with the
following command:</p>
<div class="highlight"><pre><span></span><code>virsh<span class="w"> </span>secret-set-value<span class="w"> </span>--secret<span class="w"> </span>UUID_OF_SECRET<span class="w"> </span>--base64<span class="w"> </span><span class="k">$(</span>cat<span class="w"> </span>client.volumes.key<span class="k">)</span>
</code></pre></div>
<p>If using several nova-compute nodes, the <code>secret.xml</code> file has to be
updated on the first nova-compute node (in order to ensure that the
same UUID_OF_SECRET will be used on each nova-compute node), with the
following command:</p>
<div class="highlight"><pre><span></span><code>virsh<span class="w"> </span>secret-dumpxml<span class="w"> </span>UUID_OF_SECRET<span class="w"> </span>><span class="w"> </span>secret.xml
</code></pre></div>
<p>Then with the new <code>secret.xml</code> file and the <code>client.volumes.key</code>
file, the previous operation has to be repeated on each nova-compute
node (except the first one that is already configured):</p>
<div class="highlight"><pre><span></span><code>virsh<span class="w"> </span>secret-define<span class="w"> </span>--file<span class="w"> </span>secret.xml
virsh<span class="w"> </span>secret-set-value<span class="w"> </span>--secret<span class="w"> </span>UUID_OF_SECRET<span class="w"> </span>--base64<span class="w"> </span><span class="k">$(</span>cat<span class="w"> </span>client.volumes.key<span class="k">)</span>
</code></pre></div>
<p>Finally, cinder-volume configuration files <code>/etc/cinder/cinder.conf</code>
have to be updated with the proper UUID_OF_SECRET value:</p>
<div class="highlight"><pre><span></span><code><span class="n">rbd_secret_uuid</span> <span class="o">=</span> <span class="n">UUID_OF_SECRET</span>
</code></pre></div>
<p>And cinder-volume service restarted:</p>
<div class="highlight"><pre><span></span><code>sudo<span class="w"> </span>service<span class="w"> </span>cinder-volume<span class="w"> </span>restart
</code></pre></div>
<p>After that point, any VM should be able to mount volumes using Ceph
backend!</p>